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Abstract 

This  study  investigated  whether  using  CASE  tools 
in  the  requirements  analysis  phase  of  software  development 
reduced  the  software  coding  errors  attributable  to 
requirements  analysis.  A  survey  of  the  population  gathered 
data  as  to  the  current  practices  and  efforts  in  the  use  of 
CASE  tools.  The  data  was  gathered  in  two  groups:  a  control 
group  and  a  sample  group.  The  data  was  analyzed  by  group  and 
in  various  combinations  to  obtain  a  greater  understanding 
into  the  population  trends  and  tendencies.  The  results  of 
the  research  indicate  that  the  Air  Force  does  not  use  CASE 
tools  to  any  great  extent.  Also,  error  and  cost  data  are 
not  tracked  in  a  meaningful  way  either.  The  conclusion 
drawn  from  this  research  shows  that  CASE  tool  use  is  still 
in  its  infancy  and  needs  to  begin  rapid  growth  in  order  to 
speed  up  developments  and  reduce  costs  with  the  constantly 
shrinking  budget.  The  major  recommendation  by  the  author  is 
to  perform  a  more  detailed  study  of  the  population  to 
determine  exactly  where  CASE  tools  are  being  used  and  where 
improvements  need  to  be  made. 
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REDUCING  SOFTWARE  ERRORS 


THROUGH  THE  USE  OF  COMPUTER-AIDED 
SOFTWARE  ENGINEERING  (CASE)  TOOLS 
DURING  REQUIREMENTS  ANALYSIS 


I  . _ I n  t  roduc  t i on 


Background 


Lt  Cmdr  John  Richard  Frost  stated  in  his  1984  thesis  at 


tile  Naval  Postgraduate  School: 

In  the  late  sixties  it  was  realized  that  tlie 
importance  of  software  was  rapidly  exceeding  that 
of  the  hardware  on  which  it  was  implemented.  This 
was  manifested  by  sharply  escalating  software 
costs  while  the  cost  of  hardware  underwent 
dramatic  decreases.  The  reduced  cost  of  computers 
increased  the  demand  for  them  and  hence  their 
numbers  and  the  number  and  variety  of  applications 
in  which  they  were  used  also  increased.  There  was 
a  growing  demand  for  the  ability  to  convert 
existing  applications  software  to  make  it 
executable  on  the  newer,  more  powerful,  and  less 
expensive  hardware.  The  complexity  and  size  of 
new  applications  also  increased  significantly  with 
corresponding  increases  in  the  complexity  and  size 
of  t  h  e  software  need  to  support  them.  This,  in 
turn,  led  to  a  far  greater  demand  for  software 
than  the  existing  software  industry  could  supply. 
(Frost,  1984:7) 


Today,  software  users  are  demanding  more  power  in  lap-top 


computers  than  the  overall  power  of  most  early  computers. 
Software  projects  hav<'  become  enormous  and  extremely 
complex.  "Simply  put.  the  fun<lamental  challenge  facing 
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(Phillips,  1989:64).  To  assist  the  future  software  engineer 
in  managing  this  growing  complexity,  today’s  engineers  are 
creating  automated  tools  to  simplify  the  software  management 
and  development  process.  These  products  are  called 
Computer-Aided  Software  Engineering  (CASE)  tools. 

CASE  tools  can  be  used  throughout  the  software 
lifecycle  from  conception  to  retirement.  The  research 
documented  in  this  thesis  centers  on  the  use  of  CASE  in  the 
requirements  analysis  phase. 

General  Issue 

Analysis  of  a  problem  is  never  easy.  Finding  errors 
early  in  the  design  process  can  save  thousands,  if  not 
millions,  of  dollars  (Bergland  and  others,  1990:8).  CASE 
tools  give  engineers  more  freedom  to  try  various  methods  of 
problem  analysis  in  a  shorter  period  of  time.  With  them, 
engineers  can  also  implement  requirements  changes  faster  and 
with  1  ess  cost . 

The  federal  government,  specifically  the  Department  of 
Defense  (DoD),  now  needs,  more  than  ever,  to  save  money. 

CASE  tools  can  provide  a  great  deal  of  savings  now.  As  a 
team  of  American  Telephone  and  Telegraph  (AT&T)  software 
engineers  pointed  out: 

Our  goal  in  applying  new  and  existing  technologies 
is  to  mechanize  much  of  the  front-end  process  and 
thus  reduce  development  costs  and  improve  product 
quality  (Bergland  and  others,  1990:7) 
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AT&T  has  become  one  of  the  leaders  in  CASE  tool  use  in  the 


civilian  community.  DoD  should  embrace  the  ideas  and 
concepts  of  AT&T  by  implementing  efforts  to  reduce  cost  and 
speed  up  production.  Software  development  will  cost  the 
Department  of  Defense  (DoD)  over  $30  billion  in  FY  91. 
(Ferens,  1991:4)  This  cost  translates  to  over  ten  percent 
of  the  total  DoD  budget  for  the  year.  As  software 
acquisition  grows,  the  DoD  must  find  and  use  more  cost 
effective  ways  to  develop  new  software. 

Specific  Problem 

Software  development  includes  requirements  analysis, 
design,  coding,  testing,  and  follow-on  maintenance.  This 
research  examined  the  requirements  analysis  phase  of 
software  development.  Specifically,  the  research,  described 
in  this  thesis,  focused  first  on  the  problem  of  time  and 
money  lost  as  a  result  of  errors  made  during  requirements 
analysis  conducted  without  the  assistance  of  Computer-Aided 
Software  Engineering  (CASE)  tools  and  second  on  the  effects 
tool  use  can  have  in  reducing  these  losses. 

Hypothesis  and  Investigative  Questions 

The  hypothesis  for  the  thesis  research  is  that  CASE 
tool  use  during  i- equ  i  r  emen  t  s  analysis  correlates  directly  to 
a  reduction  of  errors  attributable  to  poor  requirements 
analysis,  thereby  reducing  deficiency  and/or  error 
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correction  in  later  stages  of  a  project.  This  hypothesis 
will  be  supported  by  answering  the  following  questions; 

1.  What  percentage  of  DoD  software  developers  use  CASE 
tools  during  the  requirements  analysis  phase  of  project 
development?  What  percentage  do  not  use  CASE  tools? 

2.  For  what  reasons  did  agencies  choose  to  use  CASE  tools 
or  not  to  use  CASE  tools? 

3.  What  was  the  project  size  (lines  of  code,  cost)?  Was 
project  size  a  factor  in  deciding  whether  or  not  to  use  CASE 
tools  on  the  project? 

4.  What  were  the  initial  costs  of  using  the  tools?  Follow- 
on  or  recurring  costs? 

5.  What  cost  savings  were  found  by  using  CASE  tools  over 
not  using  CASE  tools  (including  the  recovery  of  the  initial 
cost s )  ? 

6.  What  errors  were  found?  When?  Did  using  CASE  tools 
reduce  the  number  of  errors? 

Scope  of  Research 

This  thesis  research  covered  the  use  of  CASE  tools 
during  the  requirements  analysis  phase  of  a  software 
development  project.  Which  agencies  use  CASE  tools  and  why 
users  chose  to  use  the  tools  were  questions  analyzed. 
Agencies  that  chose  not  to  use  tools  and  their  reasons  or 
lack  thereof  were  examined.  Whether  project  size  and/or 
cost  plays  a  significant  role  in  the  choice  to  use  or  not  to 
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use  CASE  tools  was  reviewed.  A  cost-benefit  analysis  on  the 
use  of  and  non-use  of  CASE  tools  was  performed.  The 
analysis  included  initial  start-up  costs,  training, 
recurring  costs,  savings  from  the  use  of  tools,  and  recovery 
of  costs. 

This  analysis  did  not  include  the  specific  CASE  tools 
being  used.  To  advocate  one  tool  over  another  is  not  the 
issue  here.  CASE  tool  use  in  later  stages  of  the  software 
development  process  (i.e.,  design,  coding,  testing,  and 
maintenance)  was  not  reviewed  in  any  detail.  These  areas 
were  highlighted  as  necessary,  however. 

The  next  chapter  reviews  current  literature  on  CASE 
tools.  The  review  concentrates  on  the  major  highlights  of 
CASE  tool  research  and  development.  Chapter  III  reviews  the 
methodology  used  to  perform  the  research  (i.e.,  population 
definition,  data  collection,  etc.).  Analysis  of  the 
collected  data  is  the  focus  of  Chapter  IV.  Chapter  V 
summarizes  findings  and  conclusions  from  the  analysis. 
Appendices  A  through  I  represent  the  survey  instrument 
development  process  and  the  data  analysis  calculations. 
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Literature  Review 


II  . 


Introduction 

Case  tools  ore  fast  becoming  the  way  to  do  business  in 

the  software  engineering  field. 

CASE  techniques  offer  a  unique  opportunity  to 
decrease  the  backlog  of  applications  development. 

They  also  present  an  opportunity  to  increase  the 
quality  of  systems  being  developed  through 
consistent  use  of  a  standard  methodology 
throughout  the  life  of  the  software.  (Batt, 

1989:6) 

Just  as  a  carpenter  needs  saws,  hammers,  drills,  and  planes, 
the  software  engineer  requires  a  tool  box  to  make  the  job 
easier  and  more  creative.  Soon,  CASE  tools  will  fill  the 
software  engineer's  toolbox.  An  understanding  of  the 
requiremert  fur  CASE  tools  and  their  continued  creation  and 
enhancement  is  needed  before  one  can  fully  appreciate  the 
importance  of  CASE  tools.  This  review  is  directed  toward  an 
understanding  of  the  software  management  problem,  the 
benefits  derived  from  the  use  of  CASE  tools,  USAF  CASE  tool 
evaluation  assistance,  USAF  CASE  tool  studies,  and  the 
direction  for  future  tool  developments. 

Sof tw are  Management  Problems 

A  software  engineer  faces  a  myriad  of  obstacles  and 
challenges  in  managing  current  software  projects. 

Complexity,  requirements  definition,  and  cost  stand  between 
the  engineer  and  successful  completion  of  a  project. 
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Today's  complexity  of  software  design  is  compares  to 

the  complexity  of  building  the  Space  Shuttle.  Many  software 

projects  "contain  several  million  lines  of  code"  (Bergland 

and  others,  1990:8).  No  longer  will  programs  of  100  to 

10,000  lines  of  code  be  adequate.  In  describing  the  tools 

of  software  engineers,  Barry  Phillips  states: 

At  the  1988  Design  Automation  Conference,  Andy 
Rappaport,  president  of  Boston’s  Technology 
Research  Group,  said  that  30%  of  embedded  systems 
ran  over  75,000  lines  of  code  -  up  from  14%  in 
1985.  What's  more,  the  average  size  of  programs 
that  run  on  internally  developed  target  hardware 
has  nearly  tripled.  (Phillips,  1989:65-66) 

Also,  learning  how  to  develop  the  complex  code  can  be  an 

education  all  by  itself.  Stuart  Feil  points  out  that: 

...the  original  developer's  tool  kit  for  the  OS/2 
operating  system  and  its  Presentations  Manager  GUI 
included  29  manuals,  130  disks  and  eight  video 
cassettes.  That's  more  than  an  evening's  reading. 
(Feil,  1989:57) 

Learning  the  OS/2  operating  system  is  equivalent  to  an 
advanced  degree  program. 

As  complex  as  software  has  become,  software  engineers 
face  other  problems.  Understanding  the  ideas  of  others  is  a 
major  stumbling  block  in  requirements  definition.  AT&T 
developers  have  found: 

Traditionally,  we  have  relied  on  natural - 1 anguage 
(e.g.  English)  documentation  as  a  communications 
medium.  However,  when  written  in  English, 
requirements  are  often  ambiguous  and  open  to 
misinterpretation.  (Bergland  and  others,  1990:9) 

Human  efforts  in  requirements  translation  lead  to  oversight 

and  costly  re-engineering  of  many  software  projects. 
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High  costs  prohibit  many  new  software  projects  or 
updates  to  older  software.  Small  firms  do  not  have  the 
resources  or  funding  needed  to  invest  in  a  major  software 
build.  Consequently,  many  small  firms  are  forced  to  operate 
with  outdated  software  that  is  difficult  to  use  and  costly 
to  maintain;  at  times,  they  operate  with  no  software  at  all. 

Benefits  of  CASE  Tools 

Even  though  the  software  engineer  faces  many  problems, 

hope  is  on  the  horizon. 

The  combined  effect  of  CASE  tools  and  sound 
development  principles  should  enable  accelerated 
software  development  and  lead  to  easing  what  has 
been  referred  to  as  a  "Software  Development 
Crisis."  (Batt,  1989:1) 

CASE  tools  can  help  avoid  the  problems  of  today’s  software 

projects  in  that  the  tools  help  do  the  job  right  the  first 

time.  As  part  of  a  special  Datamation/Price  Waterhouse 

opinion  survey,  Norman  Statland  comments: 

The  improvements  will  stem  directly  from  the 
decreased  maintenance  and  increased  productivity 
that  will  accrue  over  the  total  life  of  a  system 
developed  with  CASE  tools.  (Statland,  1989:32) 

Benefits  from  CASE  tool  use  also  include  improved  up  front 

analysis,  improved  documentation,  and  increased 

productivity . 

Analysis  of  a  problem  is  never  an  easy  task.  Finding 
errors  early  in  the  design  process  saves  thousands,  if  not 
millions,  of  dollars  (Bergland  and  others,  1990:8).  With 
CASE  tools,  engineers  are  given  more  freedom  to  try  various 
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methods  of  problem  analysis  in  a  shorter  period  of  time. 
Engineers  can  also  implement  requirements  changes  quicker 
and  with  less  cost.  An  outcome  of  improved  problem  analysis 
is  detailed  and  accurate  software  documentation. 

During  a  1987  IEEE  Conference  on  Software 
Maintenance,  a  presenter  was  quoted  as  saying  "...  system 
and  project  documentation  are  crucial  for  high  quality 
software  development  and  maintenance"  (Arthur  and  Krader, 
1989:46).  Proper  software  documentation  improves  accuracy 
and  decreases  changes  due  to  programmer  error.  El ectronic 
Design ' s  Johna  Till  describes  the  tool  Autocode  by  saying 
"With  the  new  version,  updated  documentation  can  be 
generated  each  time  code  is  created"  (Till,  1989:166). 

Every  time  an  engineer  changes  code,  the  changes  are 
automatically  documented.  The  automated  documentation  keeps 
paperwork  up-to-date  and  reduces  the  guessing  about  what  was 
changed  and  why  it  was  changed  (Till,  1989:164).  Improved 
documentation  speeds  the  production  of  actual  software  code. 

Labor  involved  in  actual  software  coding  is  one  of  the 
highest  expenses  encountered  in  the  development  process. 

With  the  use  of  CASE  tools,  best  case  reductions  of  over  99 
percent  in  coding  time  have  been  found.  A  U.S.  Department 
of  Justice  attorney/advisor  Lowell  Denning  estimates  he 
spent  between  25  and  50  hours  developing  an  introductory 
screen  without  CASE  tools.  When  he  applied  a  CASE  tool  to 
the  same  task,  he  reduced  the  development  time  to  under 
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10  minutes  (Feil,  1989:57).  Reductions  in  development  time 
mean  reductions  in  cost. 


Air  Force  CASE  Tool  Evaluation  Assistance 


Many  Air  Force  members  are  not  aware  of  the  valuable 

help  available  when  it  comes  to  evaluating  the  potential  of 

perspective  CASE  tools.  The  Software  Technology  Support 

Center  (STSC),  located  at  Hill  AFB,  Utah,  has  the  mission 

...  to  assist  Air  Force  Software  Development  and 
Support  Activities  (SDSAs)  in  the  selection  of 
technologies  that  improve  the  quality  of  their 
software  products  and  increase  the  productivity  of 
their  efforts  (  Grotzky  and  others,  1991:1). 

The  STSC  consolidates  Air  Force  support  for  selection  and 

evaluation  of  software  environments,  tools,  and  methods 

(Grotzky  and  others,  1991:1).  Because  individual  agencies 

evaluate  tools  by  different  standards  and  do  not  share  vital 

data,  agency  software  professionals  do  not  look  at  similar 

evaluations  and  thereby  waste  time  and  money.  For  these 

reasons,  the  STSC  developed  a  standard  strategy  called  the 

Test  and  Evaluation  (TSE)  Process  to  assist  in  consolidating 

the  evaluation  process. 

First,  it  gives  software  professionals  the 
information  required  to  make  intelligent  software 
tools  selections.  Second,  it  increases  the 
comparability,  consistency,  and  repeatability  of 
software  tool  evaluations.  Third,  it  improves  the 
efficiency  of  the  software  tool  evaluation  process 
through  the  reuse  of  prior  evaluation  guidelines, 
procedures,  and  data.  Fourth,  it  facilitates  the 
feedback  of  customers  requirements  to  software 
tool  developers  for  future  tool  development 
(Grotzky  and  others,  1991:2). 
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Through  this  consolidation,  all  Air  Force  agencies  can  share 
the  information  gained  to  reduce  the  cost  of  developing  and 
maintaining  software  and  increase  productivity. 

In  TSE  process  of  the  tool  selection  procedure,  the 
STSC  set  up  six  phases:  analysis,  assessment,  evaluation 
guidance,  detailed  evaluation,  recommendation,  and  selection 
(Grotzky  and  others,  1991:2-3).  This  sequence  narrows  the 
field  of  tools  being  evaluated  and  improves  the  evaluation. 

In  the  analysis  phase,  CASE  tools  available  in  one 
domain  are  identified  and  a  Long  List  of  Tools  is  generated. 
As  a  part  of  the  analysis,  characteristics  (both  quality  and 
functional)  for  the  domain  in  question  are  identified. 
Included  in  the  identification  are  the  Air  Force  essential 
characteristics  of  support  of  real-time  activities  and 
support  of  design  teams  (Grotzky  and  others,  1991:3,  16). 
The  tool  is  then  classified  into  its  particular  domain(s) 

(in  this  case,  requirements  analysis).  Finally,  a  Short 
List  of  tods  containing  Air  Force  essential  characteristics 
is  generated.  The  analysis  phase  of  the  process  is 
completed  by  STSC  personnel  with  inputs  from  academia.  Air 
Force  agencies,  and  industry  (Grotzky  and  others,  1991:3). 
As  a  part  of  the  assessment  phase,  all  of  the  identified 
characteristics  are  reviewed  to  ensure  that  all  of  the 
functional  characteristics  identified  during  analysis  have 
been  met  by  the  candidate  tool .  This  assessment  is 
completed  through  a  series  of  user  interviews,  informal 
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testing,  documentation  audits,  and  vendor  surveys.  Anyone 
can  assess  a  tool  using  STSC  guidance.  In  using  this  form 
of  assessment,  time  and  money  can  be  saved.  All  assessments 
are  combined  to  build  a  prioritized  list  used  in  the 
detailed  evaluation  phase  (Grotzky  and  others,  1991:3). 

The  evaluation  guidance  phase  activities  consist  of 
development  of  a  test  plan  and  characteristic  evaluation 
guidelines.  These  are  incorporated  in  the  "domain  Test  and 
Evaluation  Guideline  (TEG)"  (Grotzky  and  others,  1991:4). 
The  guidance  is  a  coordinated  effort  by  the  STSC  with  inputs 
from  tool  experts  and  users.  The  guidance  put  forth  in 
TEGs  is  used  in  the  detailed  evaluation  phase  as  the 
foundation  for  specific  evaluation  procedures  (Grotzky  and 
others ,  1991:4). 

Quality,  performance,  and  functionality  are  validated 

during  the  detailed  evaluation  phase.  A  sample  of  the 

quality  attributes  reviewed  are  found  in  Table  2-1  on  Page 

2-8.  The  functional  characteristics  examined  include: 

...information  capture,  methodology  support,  model 
analysis,  requirements  tracing,  data  repository, 
documentation,  data  import/export,  and  reusability 
support  (Grotzky  and  others,  1991:13). 

Development  of  tool -specif ic  Test  and  Evaluation  Procedures 

(TEPs),  executing  the  TEGs,  recording  results  and  updating 

the  characteristics,  and  furnishing  a  final  report  are  among 

the  other  activities  performed  in  this  phase.  STSC- 

supervised  evaluators  accomplish  the  detailed  review.  The 

final  report  is  compared  to  other  reports  during 


2-7 


recommendation  and  selection  phases  (Grotzky  and  others, 
1991 : 4) . 


Table  2-1.  Sample  Software  Quality  Attributes  Listing 


(Grotzky  and  others,  1991:14) 


Strengths  and  weaknesses  are  highlighted  in  the 
evaluation  phase.  Activities  include  gathering  customer- 
defined  weighting  criteria,  assigning  characteristic  weight 
defaults,  assessing  the  currency  of  the  characteristic 
listings,  calculating  tool  scores  based  on  the  weightings, 
adding  the  scores  to  the  final  reports,  and  producing  a 
comparison  report  based  on  information  from  the  preceding 
phases.  This  information  is  distributed  at  conferences,  in 
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electronic  media,  tool  reports,  and  in  newsletters  (Grotzky 
and  others,  1991:4). 

Identifying  and  comparing  tools  that  meet  the  needs  of 
a  customer  are  produced  in  the  selection  phase.  The  final 
selection  of  a  tool  is  the  responsibility  of  the  customer. 
The  STSC  is  there  as  an  advisor  only. 

As  time  has  passed,  the  STSC  has  improved  the  T&G 
process.  At  the  core  of  the  STSC's  assistance  process  is 
the  Software  Tool  Evaluation  Model  (STEM)  which  includes: 

a.  Tool  Domain 

b.  Tool  Characteristics 

1)  Air  Force  Essential  Characteristics 

2)  Default  Weights 

c.  Test  and  Evaluation  Guidelines  (TEG) 

(Grotzky  and  others,  1991:6) 

By  using  this  model,  the  STSC  and  its  customers  can  exchange 
information.  The  exchange  allows  the  customer  to  make  a 
better  decision  in  tool  selection  and  tool  comparisons 
because  of  the  improved  knowledge  base  (Grotzky  and  others, 
1991:6).  Improved  knowledge  at  little  or  no  cost  cannot  be 
ignored . 

United  States  Air  Force  Studies 

The  United  States  Air  Force  (USAF)  has  undertaken  a 
limited  number  of  CASE  tool  feasibility  studies.  These 
studies  have  been  concentrated  at  a  few  strategic  locations. 
The  Standard  Systems  Center  (SSC)  at  Gunter  AFB  AL  and  Rome 
Air  Development  Center  (RADC)  at  Griffiss  AFB  NY  are  the 
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prime  USAF  locations  for  CASE  tool  initiatives.  A  review  of 
the  major  efforts  at  these  sites  follows. 

Standard  Systems  Center  Efforts.  The  SSC  is  the  USAF 
lead  to  provide  an  integrated  CASE  tool  set.  Within  the 
past  six  months,  a  Request  for  Proposal  (RFP)  for  the 
Integrated  CASE  (ICASE)  tool  set  has  been  developed  and  is 
scheduled  for  release  in  October  1991.  The  contract,  when 
awarded,  is  estimated  to  be  worth  as  much  as  $25M.  The  RFP 
for  ICASE  requests  an  integrated  CASE  tool  set  be  provided 
for  Government  use.  The  emphasis  for  the  tool  set  is  on 
usability.  Contract  award  for  the  ICASE  contract  is 
scheduled  for  late  1991-early  1992  (Green,  1991:8). 

Rome  Air  Development  Center  Efforts.  RADC  has  a  number 
of  studies  on  CASE  tool  feasibility  ongoing.  One  such  study 
is  the  Requirements  Engineering  Testbed  undertaken  between 
July  1985  and  February  1986  (Anonymous,  1988:9). 

Requirements  Engineering  Testbed.  The  purpose  of 
the  study  was  "to  define  a  ten-year  research  and  development 
program  for  RADC's  Requirements  Engineering  Testbed  (RET)" 
(Anonymous,  1988:1).  The  primary  goal  behind  the  RET  was  to 
provide  the  means  by  which  Air  Force  users  could  test  and 
evaluate  requirements  for  future  systems.  A  secondary  goal 
was  to  promote  new  tool  use  by  industry,  as  well  as  the  Air 
Force.  RADC  developed  prototype  tools  for  the  user's 
beginning  capability.  The  prototype  also  aided  in  the 
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understanding  and  investigation  of  requirements  and  their 
implications  (Anonymous,  1988:1). 

The  problems  which  lead  to  the  RET  concept  were 
uncovered  in  an  RADC-sponsored  survey  of  mission  and 
acquisition  specialists.  From  the  survey,  three  major 
problem  areas  surfaced: 

(1)  Requirements  specifications  were  written  for 
procurers  (acquisition  engineers)  and  their 
technical  staffs.  Thus  mission  users  found  them 
to  be  too  technical  and  felt  "shut  out." 

(2)  Mission  users  and  contractors  found  it 
difficult  to  relate  A-spec  to  B-spec  (i.e.  high- 
level  system  specification  to  software 
requirements)  because  of  the  significant  "gulf" 
between  them. 

(3)  Contractors  and  mission  users  complained  that 
traceability  could  only  be  demonstrated  manually, 
making  it  hard  to  assess  requirements  coverage. 
(Anonymous,  1988:5-6) 

RADC  studies  deemed  RET  as  a  potential  solution  to  the 
problem.  The  complete  RET  model  used  two  paths  for  optimal 
results.  The  first  path  was  to  evolve  tools  and 
methodologies  like  rapid  prototyping  for  a  quick  benefit. 
Path  two  was  to  explore  high-risk  formal  requirements 
language  development  (Anonymous,  1988:2). 

The  objective  for  the  early  testbed,  in  1990,  was  to 
provide  a  number  of  requirements  engineering  capabilities 
via  integrated  tools.  Meeting  this  objective  would  support 
the  first  path  needed  for  optimization.  The  target,  for 
1995,  is  to  provide  the  integrated  capability  via  a  formal 
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requirements  language.  Doing  this  is  part  of  path  two 
(Anonymous,  1988:17). 

Work  continues  today  to  make  the  RET  a  viable  step 
toward  solving  the  requirements  definition  problems  faced  by 
the  users  and  developers. 

Direction  of  Future  CASE  Tool  Development 

With  the  creation  of  more  and  more  CASE  tools,  the 
future  looks  promising  for  improved  software  development. 
Improved  testing,  industrialization  of  development,  and 
growing  productivity  through  CASE  tools  await  the  future 
software  engineer. 

Testing  software  will  remain  a  difficult  part  of  the 
development  cycle.  To  aid  in  reducing  the  difficulties 
associated  with  testing,  software  developers  need  to  look  at 
test-generation  needs  early  in  the  design  process.  CASE 
tools,  such  as  T,  are  being  used  to  automatically  generate 
test  scenarios  during  coding.  By  developing  and  performing 
the  tests  up  front  in  the  production  process,  the  overall 
result  will  be  higher  quality,  lower  costing  projects 
(Phillips,  1989:70-71). 

Increased  tool  use  should  lead  to  an  assembly  line 
approach  of  software  development.  Software  factories,  using 
CASE  tools,  are  being  advocated  as  the  future  for  software 
engineering.  In  1985,  the  Japanese  software  community  began 
the  "Software  Industrialized  Generator  and  Maintenance  Aids 
(SIGMA)  project"  (Akima  and  Ooi ,  1989:13).  This  project 
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will  establish  a  speciali-zed  environment  for  software 
engineering  that  will  allow  cheaper,  faster,  and  larger 
software  projects  to  be  produced.  Through  the  use  of  CASE 
tools,  SIGMA  will  allow  the  mass  production  of  software 
without  the  time  and  expense  of  previous  efforts  (Akima  and 
Ooi ,  1989:14).  Industrialization  allows  for  great 
repositories  of  software  which  provide  greater  software 
reuse  possibilities.  Also,  the  industrialization  will  make 
the  production  and  combining  of  CASE  tools  cheaper  and 
easier  to  use.  Currently,  as  for  any  other  new  technology, 
CASE  tools  are  expensive.  Many  companies  make  tools  which 
are  not  readily  compatible  with  another  company’s  tools. 

So,  having  a  CASE  toolkit  has  been  impossible.  But  not  in 
the  near  future. 

Increased  productivity  is  another  predicted  outcome  of 
the  CASE  tool  revolution.  The  Datamation/Price  Waterhouse 
opinion  survey  found  the  productivity  prognosis  bright  over 
the  next  five  years. 

Fifty-one  percent  of  the  organizations  sampled 
predict  that  CASE  would  spawn  software 
productivity  jumps  of  up  to  100%  over  the  next 
five  years.  Productivity  surges  of  up  to  200%  by 
1999  are  foreseen  by  the  20%  of  the  CASE  users 
surveyed.  (Statland,  1989:33) 

Productivity  increases  of  this  magnitude  will  make  CASE 

tools  a  part  of  every  software  engineer's  development 

process.  Another  forecast  benefit  of  increased  productivity 

is  the  faster  production  and  longer  life  of  software 

developed  with  CASE  tools. 
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Fifty-four  percent  of  the  current  CASE  customers 
expect  to  see  much  quicker  software  development  by 
1993,  as  a  result  of  the  technology.  The  most 
marked  difference  between  today  and  tomorrow  comes 
in  the  maintenance  area.  A  whopping  45%  of  the 
companies  predict  that  within  five  years,  longer 
program  life  would  be  the  big  plus  of  using  CASE 
tools.  (Statland,  1989:33) 

Speeding  up  the  delivery  of  and  improving  the 

maintainability  of  new  software  would  certainly  reduce  the 

lifecycle  costs  dramatically.  A  new  day  in  software 

engineering  is  just  around  the  corner. 


Summary 

CASE  tools  provide  for  improved  software  engineering 

now  and  in  the  future.  Currently,  well-established  methods 

to  manage  the  quality  of  the  product  are  not  beneficial 

enough.  Besides  CASE  tools,  a  well-defined  software 

engineering  process  must  be  in  place  for  successful  software 

development.  This  process  must  be  understood  and  practiced 

by  every  member  of  the  organization  for  quality  and  speed  of 

CASE  tools  to  be  realized.  "Our  goal  in  applying  new  and 

existing  technologies  is  to  mechanize  much  of  the  front-end 

process  and  thus  reduce  development  costs  and  improve 

product  quality"  (Bergland  and  others,  1990:7).  Software 

is  the  future  and  improving  its  quality  while  reducing  costs 

should  be  the  objective  of  every  software  engineer. 

As  CASE  tools  become  more  prevalent  i.i  software 
development,  further  research  is  required  to 
answer  how  CASE  will  effect  the  products  delivered 
to  the  end-user.  The  end-user  should  be  able  to 
understand  whether  using  C.^.SE  will  produce  a 
higher  quality  "oftware  product,  whether  the 
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product  will  be  delivered  in  a  timely  manner,  and 
most  important  whether  the  product  will  solve  his 
business  needs  and  contribute  to  the  corporate 
strategy.  (Batt,  1989:62) 

No  longer  will  customers  tolerate  the  slow  development  ot 
the  past,  the  discovery  of  errors  after  delivery,  and  the 
unaffordable  costs  associated  with  software  development 
today.  Using  CASE  tools  will  reduce,  if  not  eliminate,  many 
of  today's  problems.  Computer-Aided  Software  Engineering 
tools  may  not  "be  the  silver  bullet  that  will  save  the 
software  industry,"  (Batt,  1989:2)  But,  when  used  w:^.,h  a 
well  defined  software  engineering  process,  they  are 
definitely  a  future  gold  mine,  not  a  passing  fad. 
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III.  Methodology 


Introduction 

The  hypothesis  underlying  the  research  is  that  CASE 
tool  use  during  re  airements  analysis  saves  money  and  time 
by  reducing  deficiency  and/or  error  correction  in  later 
stages  of  a  project.  Because  data  collection  is  vital  to 
any  causal  research,  justification,  instrumentation, 
population  and  sample  definitions,  data  collection  plans, 
statistical  tests,  and  limitations  were  established  from  the 
outset.  A  survey  was  determined  to  be  the  most  appropriate 
manner  for  gathering  the  data  inexpensively  from  a  large 
population.  A  return  of  50  percent  of  the  surveys  was 
considered  to  be  sufficient  to  provide  an  appropriate  amount 
of  data  for  analysis  (Emory,  1985:172-174).  This  chapter 
addresses  the  aforementioned  concerns. 

Justification 

CASE  tool  use  was  and  still  is  an  emerging  technology 
in  software  engineering.  Only  recently  have  CASE  tool 
studies,  of  any  significant  quantity,  been  initiated.  Of 
the  published  studies,  the  vast  majority  explored  the 
commercial  marketplace.  Government  CASE  tool  studies  are 
virtually  non-existent.  The  number  of  government  agencies 
using  tools  is  growing,  but  many  agencies  remain  skeptical 
about  CASE  tools  and  their  benefits.  This  research  should 
bridge  a  portion  of  the  believabil ity  gap. 
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Instrument 


Surveying  the  population  to  obtain  data  was  the  most 
expeditious  way  in  which  to  collect  data  without  direct 
experimentation.  Therefore,  the  survey  instrument 
development  was  vital  from  the  start. 

The  survey  originally  contained  29  pertinent  and 
unbiased  questions,  all  of  which  were  designed  to  answer  the 
research  questions  supporting  the  hypothesis.  These 
questions  were  eventually  pared  down  to  20  by  elim; noting 
redundancy  and  by  combining  related  questions. 

The  questions  were  reviewed  by  AFIT/EN  software 
engineering  experts  to  ensure  the  questions  were  applicable 
for  survey  use.  Before  sending  the  surveys  for  approval,  a 
validity  check  was  performed  to  ensure  that  the  survey 
instrument  would  not  invalidate  the  conclusions  drawn  on  the 
data  collected.  The  questions  and  survey  instrument  were 
reviewed  by  the  AFIT/EN  advisor  and  members  of  the  Graduate 
Software  Systems  Management  Program  at  AFIT. 

After  instrument  validation  and  advisor  approval,  the 
draft  instrument,  contained  in  Appendix  A,  was  sent  to 
AFIT/XP  for  approval .  AFIT/XP  then  forwarded  the  survey 
package  to  the  USAF  Manpower  and  Personnel  Center  for 
approval.  HQ  AFMPC  approved  the  survey  with  modifications 
on  20  May  91.  The  approval  letter,  outlining  the  changes, 
is  contained  in  Appendix  B.  The  final  survey  instrument 
contained  22  questions.  The  final  survey  package,  including 
cover  letter,  the  Privacy  Act  Statement,  and  the  final 
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survey  instrument,  is  presented  in  Appendix  C. 

Population/ Sample 

The  population  consists  of  United  States  Air  Force 
agencies  and  the  personnel  in  those  organizations  who 
develop  software,  along  with  a  control  group  (selected  from 
the  same  population)  of  personnel  being  trained  to  develop 
software  using  approved  software  engineering  practices.  The 
population  was  selected  through  information  gathered  from 
the  General  Services  Administration's  Federal  Software 
Management  Support  Center's  database  of  government  CASE  tool 
users.  Air  Force  Communications  Command's  (AFCC)  Computer 
Systems  Division  (CSD),  and  the  United  States  Air  Force 
(USAF)  Software  Technology  Support  Center  (STSC).  In  this 
group,  a  point-of -contact  was  identified  in  the  various 
agencies  of  the  population.  Through  these  points-of- 
contact,  mail  surveys  were  distributed,  using  a  judgmental 
process,  to  persons  managing  software  projects.  These 
people  tended  to  work  with  a  given  project  over  more  than 
one  phase  of  development  and  should  have  had  a  good 
perspective  on  the  effects,  or  lack  thereof,  that  CASE  tools 
have  had  on  a  project. 

The  population,  from  which  the  sample  and  control 
groups  were  selected,  was  approximately  20,000  (varies  due 
to  changes  in  the  development  agencies).  A  sample  size  of 
150  was  determined  to  be  adequate  to  obtain  67  responses. 

The  67  responses  were  required  to  have  a  confidence  interval 
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of  90  per  cent  plus  or  minus  10  per  cent.  This  was  derived 
from  the  equation  below: 

n(z^)p(l-p) 

(N-l)  (d^)*{z^)p(l-p) 

where 

n  =  sample  size 

N  =  20,000  (population  size) 

p  =  0.5  (maximum  sample  size  factor) 

d  =  0.1  (desired  tolerance) 

z  =  1.645  (factor  of  assurance  for  90 

percent  confidence  interval) 

A  judgment  sampling  of  the  agencies  was  taken.  The 
agencies  surveyed  include  HQ  Strategic  Air  Command,  HQ 
Tactical  Air  Command  and  subordinate  units  to  be  determined 
by  headquarters  personnel.  Air  Force  Communications  Command 
(AFCC)  CSD,  AFCC  Command  and  Control  Systems  Center,  Air 
Force  System  Command's  Aeronautical  Systems  Division,  Air 
Force  Logistics  Command's  Air  Logistics  Centers  in  San 
Antonio  TX,  Oklahoma  City  OK,  and  Sacramento  CA,  and  Air 
Force  Space  Command's  Space  and  Warning  Systems  Center. 

The  control  group  consisted  of  students  from  two 
resident  Professional  Continuing  Education  courses  at  AFIT. 
The  two  courses  were  WCSE  471  -  Software  Engineering 
Concepts  and  WCSE  472  -  Specification  of  Software  Systems. 
These  courses  teach  the  skills  needed  to  properly  develop 
requirements  and  software  specifications.  A  knowledge  of 
software  engineering  practices,  including  the  use  of  CASE 
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tools,  was  assumed  by  the  researcher.  The  students  were 
surveyed  on  the  first  day  of  the  class  to  minimize  the  bias 
of  knowledge  gained  during  the  course  of  study.  A  total  of 
six  classes  and  91  students  were  surveyed  to  develop  the 
control  group  database. 

Data  Collection  Plan 

Data  collection  consisted  of  mail  surveys.  The  mail 
survey  focused  on  statistical  data:  costs,  size,  errors, 
etc.  This  data  was  analyzed  according  to  strict  statistical 
tests  of  categorical  data.  The  tests  include  chi-squared 
distribution,  contingency  tables,  and  frequency 
distributions  of  the  population  and  control  group  responses 
to  assess  any  relationships  or  differences  (Devore, 
1987:556-591).  Tests  were  set  up  with  the  assistance  of 
AFIT/EN  Math  Department  personnel.  The  results  of  the  tests 
and  the  conclusions  drawn  from  these  tests  were  reviewed  by 
AFIT/EN  Math  and  Software  personnel . 

Statistical  Tests 

The  statistical  tests  used  to  analyze  the  data  included 
frequency  distribution  analysis,  chi-square  distributions, 
and  contingency  table  comparison  of  the  population  and 
control  groups.  The  actual  tests  performed  are  described  in 
detail  in  Chapter  4, 

Limitations 

The  survey  was  limited  to  Continental  United  States 
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(CONUS) -based  USAF  agencies  only.  Overseas  USAP  agencies 
were  not  considered  due  to  their  limited  software 
development  capabilities  and  missions.  Limited  time  for 
survey  and  analysis  of  data  forced  the  elimination  of  non- 
USAF  agencies.  Individual  CASE  tools  were  not  identified 
nor  reviewed  for  their  usability. 

Summary 

Justifying  efforts  is  a  common  practice  in  research. 
For  the  research  presented  in  this  thesis  to  succeed, 
defining  the  population,  sample,  and  control  size  were 
critical.  The  researcher  needed  to  know  the  what,  how, 
when,  and  where  of  the  data  and  the  data  collection  process 
to  develop  useful  questions  and  instrumentation.  Using  the 
appropriate  statistical  tests  made  the  research  defendable 
against  scrutiny  and  review.  The  data  analysis,  including 
the  equations  used,  is  documented  in  Chapter  4. 
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IV.  Data  Analysis 


Introduction 

In  this  chapter,  the  focus  is  on  making  sense  of  the 
data  gathered  during  the  course  of  the  research.  To  gain  an 
understanding  of  the  data,  both  the  control  group  and  sample 
data  sets  were  statistically  analyzed,  singularly  and  in 
combination.  Appendices  D  and  F  contain  the  raw  data  on  the 
control  and  population  samples,  respectively.  Appendices  E, 
G,  H,  and  I  contain  the  statistical  analyses  results 
mentioned  above.  All  analyses  are  based  on  meeting  a  90 
percent  confidence  interval . 

Control  Group  Findings 

Ninety-one  cases  were  used  in  defining  the  control 
group.  Not  all  survey  respondents  completed  every  question. 
The  number  of  questions  left  unanswered  caused  the 
confidence  in  the  data  to  fall  below  a  90  percent  confidence 
level,  making  valid  statistical  analysis  impractical.  These 
questions  will  be  discussed  as  required. 

Frequency  Distributions.  By  taking  frequency 
distributions  for  each  survey  question,  the  validity  of  the 
data  (meeting  the  90  percent  confidence  interval  criteria) 
can  be  assessed.  Because  respondents  could  provide  more 
than  one  answer,  questions  3,  9,  and  18  are  not  analyzed 
with  frequency  distributions.  Instead,  the  totals  are 
displayed  in  table  format. 
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Rank .  Ninety  of  91  respondents  answered  the 


question  concerning  rank.  As  Figure  E-1  shows,  a  majority 
of  the  respondents  fell  into  the  01  to  03  and  CS-10  to  GS-12 
categories.  This  characteristic  is  consistent  with  the 
working  levels  of  most  Air  Force  units. 

Programming  Experience.  In  Figure  E-2,  90  of  91 
respondents  answered  the  question  concerning  software 
programming  experience.  The  respondents  were  normally 
distributed  over  the  five  categories.  Three  to  6  years  of 
experience  was  the  most  common  (at  30.8  percent)  response. 

Type  of  Systems  Developed.  Respondents  could 
provide  multiple  answers  about  the  type  of  systems  developed 
in  their  offices,  depending  on  the  mission  of  their 
organization.  104  responses  were  obtained.  Command  and 
control,  avionics,  management  information,  and  mission 
planning  systems  were  the  most  widely  developed  software 
systems . 


Table  4-1.  Systems  Under  Development 


No  Response 

6 

Command  and  Control 

20 

Intel  1 igence 

5 

Avionics 

17 

Management  Information 

14 

Other  Embedded/Real -Time 

19 

Other 

23 
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Nvunber  of  Systems  Being  Developed.  Figure  E-3 


shows  that  5  of  the  91  respondents  did  not  answer  the 
question  covering  how  systems  their  offices  currently  had 
under  development.  A  majority  (34.1  percent)  of  the 
surveyed  offices  develop  1  to  2  systems  at  present.  Twenty- 
one  of  the  offices  are  developing  10  or  more  systems. 
Seventeen  offices  have  no  systems  development  underway. 

These  offices  develop  software  policy,  provide  maintenance, 
or  test  software. 

Cost  of  Systems  Being  Developed.  As  Figure  E-4 
shows,  the  clear  majority  of  systems  under  development  cost 
$1  million  or  more.  One  fact  stood  out  in  the  answers 
concerning  average  cost  of  development:  many  people  are  not 
aware  of  the  costs  of  systems  their  offices  are  developing. 

System  Lines  of  Code.  Again,  the  responses  held 
consistent  with  the  vast  majority  of  major  systems 
development.  As  shown  in  Figure  E-5,  33  percent  of  the 
systems  being  developed  contained  100,000  or  more  lines  of 
code . 

Up  until  this  point  in  the  analysis,  the  results  were 
used  to  provide  an  insight  into  the  respondents’ 
backgrounds,  their  organizations  (in  general),  and  system 
development  activities.  From  this  point  on,  the  focus  will 
be  on  CASE  tool  use  by  personnel  and  organizations. 

CASE  Tool  Use  by  Organizations.  Sixteen  of  the  91 
people,  compared  in  Figure  E-6,  surveyed  did  not  respond  as 
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to  whether  or  not  their  organizations  use  CASE  tools. 
Forty-one  of  75  (54.7  percent)  of  the  respondents  answered 
that  their  organizations  do  not  use  CASE  tools  at  all. 
Organizations  that  use  CASE  tools  account  for  45.3  percent 
of  the  total.  The  literature  reviewed  in  Chapter  2  suggests 
that  the  author  expected  a  higher  percentage  of  offices  that 
do  not  use  CASE  tools.  This  almost  even  split  of  use  and 
non-use  seems  to  indicate  that  either  the  Air  Force  is  a 
leader  in  CASE  tool  use  or  that  the  control  group  offices 
selected  were  high  in  the  number  of  CASE  tool  users.  This 
divergence  from  published  literature  will  be  analyzed 
further  when  the  comparison  is  made  between  the  control 
group  and  the  population  sample  later  in  this  chapter. 

Personal  CASE  Tool  Users.  Of  84  responses  shown 
in  Figure  E-7,  62  or  73.8  percent  do  not  use  CASE  tools  at 
all.  This  fact  is  constant  with  the  findings  of  the 
literature  search. 

Areas  of  CASE  Tool  Use.  Again,  respondents  could 
provide  more  than  one  answer  to  the  question  concerning  how 
their  organizations  used  CASE  tools  in  the  software 
lifecycle.  One  hundred  fifteen  answers  were  given  to  the 
question.  33  percent  or  38  people  responded  that  their 
organization  does  not  use  CASE  tools.  Twenty-two  answers 
appeared  in  the  "Do  not  know”  category.  Of  the 
organizations  using  CASE  tools,  requirements  analysis  and 
design,  with  17  and  14  responses  respectively,  were  the  top 
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areas  of  use.  This  is  steady  with  the  findings  of  the 
literature  reviewed.  As  Chapter  2  points  out,  lower  CASE 
tools  are  just  being  developed  for  coding,  testing,  and 
maintenance.  The  upper  CASE  tools  for  requirements  analysis 
and  design  are  being  used  regularly  by  software  development 
organizations  (Bartow  and  Key,  1991:3-4).  Twenty-three 
persons  did  not  respond  at  all  to  the  question.  This  number 
was  not  included  in  the  115  mentioned  earlier  in  the 
paragraph. 


Table  4-2.  Areas  of  CASE  Tool  Use 


No  Response 

23 

Requirements  Analysis 

17 

Design 

14 

Coding 

8 

Testing 

9 

Maintenance 

7 

None 

38 

Do  Not  Know 

22 

Requirements  analysis  garnered  only  17  total  responses. 
This  low  number  indicates  that  the  probability  exists  that 
CASE  tool  use  for  requirements  analysis  in  the  USAF  is  low, 
as  was  expected  by  the  author. 

Initial  CASE  Tool  Cost.  Figure  E-8  shows  the 
breakout  of  responses  received  concerning  initial  costs  of 
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CASE  tool  use.  Forty-three  people,  or  47.3  percent  of  the 
responses,  indicated  that  the  initial  costs  of  CASE  tool  use 
were  unknown.  Only  13  respondents  could  provide  a  genuine 
categorical  answer.  Of  the  15  responses  in  the  $0-999 
category,  12  answers  could  be  traced  back  to  where  the 
respondents  had  signified  that  their  organization  did  not 
use  CASE  tools.  Also,  the  20  "no"  responses  could  be 
attributed  to  no  organizational  use  of  CASE  tools. 

Recurring  CASE  Tool  Costs.  The  numbers  and 
relationships  described  above  are  almost  repeatable  in  this 
section.  The  distribution  in  Figure  E-9  shows  that  of  the 
22  "no"  responses  and  17  "$0-999"  category  responses,  all 
but  3  of  the  39  were  due  to  no  organizational  use  of  CASE 
tools.  Four  of  the  12  persons  who  could  identify  initial 
cost  categories  did  not  know  the  recurring  cost  of  CASE  tool 
use.  48.4  percent,  or  48,  of  the  respondents  did  not  answer 
for  either  because  they  lacked  of  knowledge  or  did  not  use 
CASE  tools. 

First  Exposure  to  CASE  Tools.  Nineteen  persons 
gave  no  response  concerning  their  first  exposure  to  CASE 
tools.  The  remaining  72  responses  displayed  in  Figure  E-10, 
broke  out  in  this  fashion:  the  majority  learned  from 
magazines/ journals  and  the  job,  23  and  21  respectively;  16 
persons  first  learned  of  CASE  tools  in  school;  the  remaining 
12  persons  were  first  introduced  to  CASE  tools  from  other 
sources . 
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Areas  Lacking  Enough  Data  for  Statistical 


Significance.  As  presented  in  Figures  E-11  through  E-19,  a 
lack  of  response,  indicated  in  the  "no"  response  categories, 
by  the  control  group  made  statistical  evaluation  of  areas 
identified  in  Table  4-3  impractical.  Respondents  gave  no 
response  primarily  because  of  organization  non-use  of  CASE 
tools.  Another  reason  for  no  answer  was  that  the  questions 
were  ignored. 

Table  4-3.  Areas  of  Impractical  Statistical  Evaluation 

CASE  Tool  Experience  Levels 

Requirements  Analysis  Errors/KLOC  without  CASE  Tool  Use 

Total  Error  Percentage  Attributable  to  Requirements 

Analysis  without  CASE  Tool  Use 

Requirements  Analysis  Errors/KLOC  with  CASE  Tool  Use 

Total  Error  Percentage  Attributable  to  Requirements 

Analysis  with  CASE  Tool  Use 

CASE  Tool  Use  Increased  Requirements  Analysis  Error  Rates 

CASE  Tool  Use  Increased  Productivity 

C.ASE  Tool  Use  Increased  Development  Costs 

CASE  Tool  Use  Increased  Project  Completion  Time 


Sample  Group  Findings 

The  sample  group  was  defined  by  67  total  responses.  Again, 
not  all  respondents  completed  every  question.  Questions 
which  are  statistically  insignificant  (well  below  a  90 
percent  confidence  interval)  are  identified  and  the 
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implications  discussed. 


Frequency  Distributions.  As  was  the  case  with  the 
control  group,  the  sample  group  responses  were  reviewed 
through  the  use  of  frequency  distribution  charts.  This 
review  was  undertaken  to  insure  the  validity  of  the  data 
being  analyzed  and  to  gain  an  insight  into  the  tendencies  of 
the  sample  group. 

Rank .  All  67  respondents  provided  answers 
concerning  rank.  Figure  G-1  shows  that  37  (55.2  percent')  of 
the  respondents  were  in  the  01-03  category.  The 
concentration  of  the  response  is  consistent  with  that  of  the 
control  group. 

Programming  Experience.  As  with  rank,  responses 
were  received  from  all  67  persons  concerning  their 
programming  experience.  Seventeen  respondents  had  software 
experience  levels  of  10  or  more  years.  67.8  percent  (46 
persons)  had  at  least  3  years  experience.  The  distribution 
of  responses  in  contained  in  Figure  G-2. 

Type  of  Systems  Developed.  Eighty  responses  were 
obtained  to  this  multiple  answer  category.  As  Table  4-4  on 
the  next  page  shows,  management  information  and  command  and 
control  systems  were  the  systems  found  to  be  under 
development  most  often. 

Number  of  Systems  Being  Developed.  Figure  G-3 
indicates  that  all  of  the  sample  group  answered  the  question 
pertaining  to  the  number  of  systems  under  development.  55.2 
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Table  4-4.  Systems  Under  Development 


No  response 

5 

Command  and  Control 

19 

Intel  1 igence 

10 

Avionics 

11 

Management  Information 

22 

Other  Embedded/Real -Time 

6 

Other 

7 

percent  of  the  respondents  indicated  that  their  office  had  1 
to  2  systems  in  development  at  the  present  time.  15  offices 
surveyed  have  no  systems  under  development.  Once  again, 
these  offices  develop  software  policy,  provide  maintenance, 
or  test  software. 

Cost  of  Systems  Being  Developed.  As  was  the  case 
with  the  control  group,  the  cost  of  a  majority  of  the 
systems  under  development  in  the  sample  group  is  over  $1 
million  (19  responses).  In  Figure  G-4,  one  can  see  that  13 
persons  marked  that  their  costs  are  $0-999.  This  fact 
strongly  suggests  that  two  of  the  offices  in  the  previous 
category  do  develop  systems  .  However  these  offices  are 
between  development  projects. 

System  Lines  of  Code.  In  the  lines  of  code 
category,  66  of  67  persons  responded  with  respect  to  system 
lines  of  code  (LOC)  for  development  projects.  Although  this 
is  lower  than  the  90  percent  confidence  interval,  meaningful 
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conclusions  can  be  drawn  from  the  data.  Twenty-nine 
respondents,  or  43.3  percent,  indicated  that  their  projects 
produced  10,000  to  99,999  LOG,  as  depicted  in  Figure  G-5. 
This  ranked  first  in  this  category  as  compared  to  the 
control  group  category  where  systems  of  over  1  million  LOG 
were  more  prevalent. 

As  in  the  previous  section,  the  preceding  six 
paragraphs  provided  a  background  into  the  survey 
respondents.  The  remainder  of  this  section  is  targeted  at 
gaining  an  insight  into  GASE  tool  use  in  the  sample  group. 

GASE  Tool  Use  Bv  Organizations.  Figure  G-6 
exhibits  the  results  of  the  survey  responses  on  case  tool 
use  by  organizations.  As  with  the  control  group,  the 
respondents  in  the  sample  group  indicated  that  more  than  50 
percent  of  the  surveyed  organizations  use  GASE  tools.  Again 
this  percentage  was  greater  than  my  initial  supposition  had 
anticipated  the  percentage  to  be.  The  52.2  percent  of 
organizations  that  use  GASE  tools  is  even  higher  than  the 
control  group's  45.3  percent.  The  use  of  GASE  tools  in 
United  States  Air  Force  software  development  organizations 
is  escalating  or  the  offices  surveyed  had  a  higher  CASE  use 
rate  as  was  indicated  in  the  control  group  on  page  4-4. 

Personal  CASE  Tool  Use.  Of  the  67  persons 
responding  in  this  category  (Figure  G-7),  48  persons  (71,6 
percent)  do  not  personally  use  CASE  tools.  This  is 
consistent  with  the  control  group's  73.8  percent  lack  of 
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personal  CASE  tool  use. 

Areas  of  CASE  Tool  Use.  The  respondents  could 

provide  multiple  answers  pertaining  to  where  their 

organizations  used  CASE  tools  in  the  software  development 

lifecycle.  A  total  of  139  answers  were  received  in  this 

* 

category.  32  persons  responded  that  their  organization  did 
not  use  CASE  tools.  8  persons  failed  to  answer  the  question 
and  6  did  not  know  where  CASE  tools  were  used  in  their 
organizations.  Of  the  remaining  21  respondents,  they 
accounted  for  the  93  remaining  answers  in  the  category. 
Looking  at  this  factor,  one  can  find  the  indication  that 
when  an  organization  uses  CASE  tools,  tools  are  used  in  more 
than  one  part  of  the  software  development  lifecycle.  The 
breakout  of  all  responses  in  this  category  can  be  seen  in 
Table  4-5  on  the  next  page.  Requirements  analysis  received 
only  16  responses  in  the  sample  group.  From  this  total,  it 
is  clear  that  CASE  tool  use  in  the  area  of  requirements 
analysis  is  low  in  the  USAF. 

First  Exposure  to  CASE  Tools.  63  of  67  survey 
respondents  answered  the  question  concerning  their  first 
exposure  to  CASE  tools.  Again,  this  level  is  less  thai*  the 
90  percent  confidence  interval,  but  significant  enough  to 
provide  a  meaningful  trend.  Of  the  63  responses  in  this 
category,  job  and  school  represented  67.1  percent  (45  total) 
of  the  answers.  This  is  different  than  the  control  group 
where  job  and  magazines  and  journals  took  the  majority 
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Table  4-5.  Areas  of  CASE  Tool  Use 


No  Response 

9 

Requirements  Analysis 

16 

Design 

20 

Coding 

« 

14 

Testing 

8 

Maintenance 

16 

None 

32 

Do  Not  Know 

6 

responses  were  analyzed  using  frequency  distribution  charts. 
This  combining  of  data  allowed  a  better  view  of  the  software 
requirements/development  population  as  a  whole  and  opened  to 
share.  Figure  G-10  presents  the  sample  group  responses. 

Areas  Lacking  Enough  Data  for  Statistical 
Significance.  The  number  of  respondents  who  failed  to 
respond  (  tabulated  under  "no  response"  )  made  a  number  of 
categories  made  statistical  analysis  meaningless.  Figures 
G-8,  G-9,  and  G-11  to  G-19  exhibit  the  results  of  the  survey 
responses.  Table  4-6,  on  page  4-13,  shows  the  categories 
where  meaningful  statistical  analysis  is  unrealistic. 

Combined  Analysis  of  Groups 

Frequency  Distributions.  Combining  the  control  group 
and  sample  group  data  to  look  at  the  population  as  a  whole 
is  the  purpose  of  this  section  of  the  chapter.  158  survey 
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Table  4-6.  Areas  of  Impractical  Statistical  Evaluation 

Initial  Cost  of  CASE  Tool  Use 
Recurring  Cost  of  CASE  Tool  Use 
Length  Person  Has  Used  CASE  Tools 

Requirements  Analysis  Errors/KLOC  without  CASE  Tool  Use 

Total  Error  Percentage  Attributable  to  Requirements 
Analysis  without  CASE  Tool  Use 

Requirements  Analysis  Error/KLOC  with  CASE  Tool  Use 

Total  Error  Percentage  Attributable  to  Requirements 
Analysis  with  CASE  Tool  Use 

CASE  Tool  Use  Increased  Requirements  Analysis  Error  Rates 

CASE  Tool  Use  Increased  Productivity 

CASE  Tool  Use  Increased  Development  Costs 

CASE  Tool  Use  Increased  Project  Completion  Times _ 


analysis,  many  areas  in  which  it  was  previously  impractical 
to  do  so.  Questions  3,  9,  and  18  will  not  dg  reviewed  in 
this  section  because  reviewing  them  would  be  repetitious. 
With  this  data  combination,  17  of  19  areas  of  review  can  be 
analyzed  to  the  needed  population  understanding. 

Rank .  As  Figure  G-1  shows,  the  01  to  03  category 
was  the  area  in  which  most  respondents  fell  over  any  other 
category.  Sixty-seven,  or  42.4  percent,  of  the  respondents 
were  in  this  category.  GS-10  to  GS-12  was  the  second 
highest  area  of  respondents  with  29.7  percent  (47 
respondents)  marking  this  category.  With  the  majority  of 
the  respondents  and  the  majority  of  Air  Force  software 
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personnel  working  levels  in  these  two  categories,  it  can  be 
determined  that  the  survey  was  distributed  to  the 
appropriate  people.  In  the  rank  and  grade  data,  no 
anomalies  were  foiuid. 

Programming  Experience.  In  reviewing  the 
experience  data,  the  area  of  3  to  6  years  received  the 
highest  response  rate.  This  is  consistent  with  the 
experience  levels  of  the  categories  focused  on  in  the  rank 
data  and  was  consistent  with  the  author's  expected  findings. 
In  looking  at  Figure  H-2,  one  can  note  the  almost  perfect 
normal  distribution  of  the  responses  in  the  chart. 

Number  of  Systems  Being  Developed.  In  a  majority 
of  the  respondents'  offices,  there  are  only  1  to  2  systems 
being  developed  at  survey  completion  time.  In  Figure  H-3, 
one  can  notice  that  68  responses  were  received  in  this  area 
as  compared  to  the  next  highest  response  area  which  was  no 
systems  under  development.  Only  33.5  percent  of  the 
respondents  indicted  that  their  organizations  had  3  or  more 
systems  in  development  at  the  time  of  survey  completion. 

The  number  of  large  software  development  organizations  in 
the  Air  Force  is  not  as  high  as  the  50  percent  the  author 
anticipated  during  this  effort. 

Cost  of  Systems  Being  Developed.  Ninety-one  of 
the  158  responses  (Figure  H-4)  indicated  that  the  systems 
under  development  in  their  organizations  had  costs  of  over 
$100K.  Of  these  91,  61  persons  marked  that  system  costs 
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were  $1M  or  more.  With  software  costs  being  one  of,  if  not, 
the  most  expensive  portions  of  program  development  the 
statement  in  Chapter  I  concerning  the  fact  that  software 
cost  DoD  over  $30  billion  in  the  FY91  Budget  is  not  out  of 
line  or  realm  of  comprehension  (Ferens,  1991:4). 

Systems  Lines  of  Code.  The  data,  in  Figure  H-5, 
exhibits  that  the  majority  of  systems  under  development 
during  this  survey  had  more  than  10  KLOC .  Forty-nine 
persons  responded  that  their  development  systems  had  between 
10,000  and  99,999  lines  of  code.  Forty-six  responses 
indicated  that  development  systems  contained  more  than 
100,000  LOC .  These  numbers  are  consist  with  the  current 
trends  toward  software  project  size. 

As  was  the  case  with  the  group  and  sample  sections.  The 
previous  six  paragraphs  have  built  a  basis  for  understanding 
the  population's  makeup.  This  profile  sets  the  general 
population  as  being  at  the  company  grade  level  or  civilian 
equivalent,  with  3  to  6  years  experience  in  software 
programming,  developing  a  variety  of  systems,  normally 
develop  1  to  2  systems  at  one  time,  system  costs  are  $100K 
or  greater,  and  contain  at  least  10,000  LOC.  The  next  13 
sections  will  build  the  population  profile  in  regards  to 
their  use  of  CASE  tools. 

CASE  Tool  Use  by  Organizations.  The  Figure  H-6 
distribution  of  CASE  tool  use  by  organizations  is  almost 
even  with  non-users  holding  a  slim  72  to  69  advantage.  The 
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author,  through  reading,  experience,  and  research,  expected 
only  25  percent  of  the  organizations  to  use  CASE  tools. 

This  trend  is  considered  significant  because  Air  Force 
organizations  are  using  or  have  the  ability  to  use  CASE 
tools  in  their  software  lifecycle. 

Personal  CASE  Tool  Users.  The  trend  exhibited  in 
Figure  H-7  signifies  that  even  though  the  organizations  are 
using  CASE  tools,  the  individuals  are  not  doing  so.  Sixty- 
nine  percent  (109  respondents)  marked  that  they  do  not  use 
CASE  tools.  This  fact,  in  connection  with  the  previous 
paragraph,  lends  credence  to  the  assumption  that 
organizations  have  the  tools  available  but  the  people  are 
not  using  them.  Reasons  for  not  using  the  tools  could  be: 
too  difficult  to  learn  or  understand;  being  set  in  one's 
ways;  afraid  of  using  tools;  no  encouragement  to  use  tools. 

Initial  CASE  Tool  Cost.  Seventy-seven  persons,  or 
48.7  percent,  responded  that  they  did  not  know  the  costs 
associated  with  the  acquisition  of  CASE  tools  for  their 
organizations.  This  high  percentage,  seen  in  Figure  H-8,  is 
indicative  of  the  fact  that  the  organizations  surveyed 
either  do  not  keep  records  of  the  costs  or  that  the  costs 
are  not  known  throughout  the  organization. 

Recurring  CASE  Tool  Costs.  As  was  the  situation 
described  in  reference  to  initial  cost,  the  same  can  be  said 
about  the  recurring  CASE  tool  costs.  Figure  H-9  portrays 
the  identical  responses  and  percentages  for  those  persons 
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not  knowing  the  costs.  The  reasons  behind  this  trend  are 
the  same  ones  stated  previously. 

First  Exposure  to  CASE  Tools.  The  persons 
surveyed  had  a  wide  range  of  ways  in  which  they  were  exposed 
to  CASE  tools.  On  the  job  garnered  45  responses  while 
school  and  magazines  and  journals  gained  37  and  34  responses 
respectively.  Figure  H-10  displays  the  data.  Other  sources 
were  marked  by  17  persons  without  any  indications  as  to  what 
those  sources  could  be.  With  this  information,  one  can 
extrapolate  that  CASE  tools  are  becoming  of  increasing 
interest  to  the  software  community  on  the  whole. 

Experience  with  CASE  Tools.  When  the  control  and 
sample  groups  were  examined  individually,  this  area  of  the 
survey  could  not  be  reviewed  because  of  the  number  of  no 
responses.  However,  by  combining  the  data,  an  assessment  of 
the  population's  experience  using  CASE  tools  can  be 
inferred.  With  this  data  combination.  Figure  H-11  presents 
the  fact  that  experience  levels  for  those  surveyed  is  less 
than  1  year.  This  factor  is  a  result  of  the  newness  of  CASE 
tool  technology. 

Errors  Attributable  to  Requirements  Analysis 
without  CASE  Tool  Use.  In  this  area,  the  158  respondents 
had  a  wide  range  of  answers.  But  as  shown  in  Figure  H-12, 
no  response  answers  lead  the  pack  with  82  (51.9  percent) 
total.  A  majority  of  the  no  response  answers  are  tracked  to 
organizations  that  do  not  use  CASE  tools.  Of  the  remaining 
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responses,  31  persons  indicated  that  less  than  10 
errors/KLOC  were  traceable  to  requirements  analysis  (RA) ,  20 
indicated  that  10  to  25  errors/KLOC  were  traceable  to  RA,  12 
indicated  that  26  to  50  errors/KLOC  traceable  to  RA,  10 
indicated  that  51  to  100  errors/KLOC  traceable  to  RA,  and  3 

I 

indicated  that  over  100  errors/KLOC  were  traceable  to  RA. 
Many  comments  from  respondents  indicated  that  their 
organizations  do  not  track  errors  to  specific  areas.  These 
comments  demonstrate  that  Air  Force  software  development 
organizations  need  to  track  errors  closer  to  find  trends  and 
begin  to  eliminate  problem  areas. 

Percentage  of  Total  Errors  Credited  to 
Requirements  Analysis.  As  in  the  previous  data  set,  no 
responses  from  organizations  not  using  CASE  tools  took  the 
majority  share  of  responses.  In  Figure  H-13,  the  percentage 
of  total  errors  credited  to  requirements  analysis  varied 
form  less  than  5  percent  to  over  50  percent.  This  data  set 
suffers  the  same  lack  of  error  tracking  problems  that  the 
previous  data  set  encountered. 

In  the  next  four  paragraphs,  insights  into  the 
population's  views  of  CASE  tools  were  examined.  The  opinions 
of  the  population  surveyed  can  be  expanded  to  the  entire  Ar 
Force  software  development  arena. 

Errors  Increased  While  Using  CASE  Tools. 

Excluding  the  78  "no"  responses,  the  majority  of  the  answers 
(33  total)  indicate  that  the  population  neither  agree  or 
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disagrees  that  error  rates  increased  when  CASE  tools  were 
used  in  software  development.  Forty-three  persons  responded 
that  they  either  disagreed  or  strongly  disagreed  with  the 
statement  as  seen  in  Figure  H-16.  These  disagreement  leads 
to  the  conclusion  that  the  population  understands  the 
purpose  of  CASE  tools. 

Productivity  Increased  Using  CASE  Tools.  Forty 
persons  responded  that  they  agreed  or  strongly  agreed  that 
CASE  tools  increased  productivity.  Another  40  had  no 
opinion  one  way  or  another  concerning  the  statement.  In 
Figure  H-17,  76  answers  are  shown  to  have  been  no  responses. 

Development  Costs  Increased  Using  CASE  Tools. 
Figure  H-18  displays  the  breakout  of  answers  in  this  area  of 
analysis.  Seventy-nine  "no”  responses  can  be  found. 
Twenty-five  persons  indicated  they  disagreed  with  the 
statement.  However,  52  persons  indicated  that  they  agree  or 
had  no  opinion  that  overall  development  costs  increased 
using  CASE  tools.  This  is  opposite  of  the  intended  purpose 
of  CASE  tools  to  lower  development  costs.  Many  respondents 
did  comment  that  their  opinions  were  based  on  the  first  or 
only  project  completed  in  their  organization  to  use  CASE 
tools . 

Project  Completion  Time  Increased  Using  CASE 
Tools .  The  opposite  of  the  intended  purpose  of  CASE  tools 
is  displayed  in  this  area  of  inquiry.  In  Figure  H-19,  "no" 
responses  tallied  79,  disagreed  tallied  33,  and  44  agreed  or 
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had  no  opinion  that  using  CASE  tools  increased  project 
completion  time.  CASE  tools  are  designed  to  reduce  the  time 
needed  to  complete  projects.  A  number  of  respondents  stated 
that  the  increased  completion  time  was  a  result  of  learning 
to  use  the  tools  or  not  knowing  how  to  use  the  tools  at  all. 

With  the  result  in  the  last  two  areas,  it  is  clear  that  more 
education  is  needed  into  the  purpose  of  CASE  tools. 

Areas  Still  Lacking  Enough  Data  for  Statistical 
Significance .  The  areas  of  errors/KLOC  while  using  CASE 
tools  and  percentage  of  total  errors  credited  to 
requirements  analysis  using  CASE  tools,  even  aftei 
combination  of  data  from  the  two  groups,  did  not  have  enough 
data  to  be  analyzed.  The  lack  of  response  resulted  from  a  ^ 

lack  of  data  tracked  in  these  areas,  newness  of  CASE  tools 
to  organizations,  and  non-tool  use. 

Contingency  Tables  and  Chi-Square  Calculations.  The 
contingency  tables  and  Chi-Square  calculations  completed  in 
Appendix  I  are  explained  in  this  section.  The  tables  and 
calculations  were  completed  using  the  SAS  Statistical 
program  on  the  AFIT  computer  cluster.  The  conclusions  drawn 
from  these  statistics  were  confirmed  with  APIT/EN 
statistical  experts.  The  formula  and  reasoning  behind  the 
Chi-Square  is  as  follows: 
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Null  hypothesis:  The  two  variables  are  independent. 
Alternative  hypothesis;  The  two  variables  are  dependent. 
Rejection  region:  Reject  Hg  if  Chi-Square  exceeds  the 

tabulated  critical  value  for  a  equals 
alpha  and  df  =(r-l)(c-l),  where 
r  =  number  of  rows  in  the  table 

c  =  the  number  of  columns  in  the  table  (Ott,  1977:294-298). 

The  definition  for  independence  is: 

Two  variables  that  have  been  categorized  in  a  two- 
way  table  are  independent  if  the  probability  that 
a  measurement  is  classified  into  a  given  cell  of 
the  table  is  equal  to  the  probability  of  being 
classified  into  that  column.  This  must  be  true 
for  all  cells  of  the  table  (Ott,  1977:295) 

Independent  is  stating  that  the  two  groups  are  not  related 

in  any  way.  Dependent  is  just  the  opposite;  the  control  and 

sample  groups  are  related.  Cells  are  possible  responses. 

The  observed  and  expected  cell  counts  and  calculations  for 

each  data  source  are  contained  in  Appendix  I,  Tables  I-l 

through  1-19. 

Programming  Experience.  The  Chi-Square  value 
calculated  for  experience  was  4.935.  This  number  is  less 
than  the  critical  value  of  9.23635.  Therefore,  we  accept 
the  null  hypothesis  that  the  control  and  sample  groups  are 
independent . 

Cost  of  Systems  Being  Developed.  The  Chi-Square 
value  is  26.415.  Since  this  is  greater  than  the  critical 
value  of  9.23635,  we  reject  the  null  hypothesis  and 
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determine  that  the  control  and  sample  groups  are  dependent. 
The  cell  values,  or  observations,  have  to  much  variance  too 
to  be  acceptable. 

System  Lines  of  Code.  The  Chi-Square  value  of 
14.856  is  greater  than  the  critical  value  of  9.23635. 
Therefore,  we  reject  the  null  hypothesis  and  can  determine 
that  the  control  and  sample  groups  are  dependent. 

CASE  Tool  Use  by  Organizations.  The  critical 
value  of  4.60517  is  less  than  the  Chi-Square  value  of 
13.797.  The  null  hypothesis  is  rejected  and  the  control  and 
sample  groups  are  deemed  to  be  dependent. 

Personal  CASE  Tool  Use.  The  values  for  the 
critical  and  Chi-Square  number*  are  the  same  as  the 
organizational  data.  Therefore,  the  same  conclusions  and 
reasoning  applies  to  this  data. 

First  Exposure  to  CASE  Tools.  The  Chi-Square 
value  for  this  data  is  13.027.  This  exceeds  the  critical 
value  of  7.77944.  The  null  hypothesis  is  rejected  and  the 
control  and  sample  groups  are  determined  to  be  dependent. 

Percentage  of  Errors  Credited  to  Requirements 
Analysis  without  CASE  Tools.  The  critical  value  of  9.23635 
is  exceeded  by  the  Chi-Square  value  of  23.499.  Therefore, 
the  cell  counts  vary  too  much  to  be  accurate  and  the  null 
hypothesis  is  rejected.  Thus  the  control  and  sample  groups 
are  dependent. 

Areas  of  Uncertainty.  The  11  areas  not  discussed 
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above  had  problems  in  the  application  of  the  Chi-Square 
distribution.  In  all  cases,  cell  counts  less  than  5  in  at 
least  30  percent  of  the  cells  made  the  validity  of  the  Chi- 
Square  calculations  questionable. 

Summary 

A  great  deal  of  data  was  collected  and  analyzed  during 
the  course  of  this  work.  In  many  cases,  the  data  lacked  the 
necessary  numbers  for  a  good  insight  into  the  population. 
Answers  were  missing  in  many  categories.  On  the  whole, 
however,  the  data  did  give  us  enough  information  to 
determine  if  we  can  answer  the  research  questions  listed  in 
Chapter  I.  Chapter  V  will  provide  t^ose  answers  and  make 
the  conclusions  and  recommendations  that  have  occurred  as  a 
result  of  this  research. 
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V.  Conclusions  and  Recommendations 


Introduction 

This  chapter  wraps  up  the  research  into  CASE  tool  use 
in  requirements  analysis  reducing  software  errors  associated 
with  requirements  analysis.  Conclusions  and  answers  to  the 
research  questions,  along  with  recommendations  for  future 
research  into  this  area,  are  provided. 

Conclusions 

Research  Question  #1.  The  question  was:  What 
percentage  of  DoD  software  developers  use  CASE  tools  during 
the  requirements  analysis  phase  of  project  development? 

What  percentage  do  not  use  CASE  tools?  • 

The  data  analyzed  in  Chapter  IV  indicates  that  the 
majority  of  software  developers  do  not  use  CASE  tools  in  the 
requirements  analysis  phase  of  project  development.  This  is 
consistent  with  the  findings  in  the  literature  review.  The 
Air  Force  is  just  beginning  to  enter  the  CASE  tool  arena  in 
the  requirements  analysis  area.  Within  the  next  10  years, 
one  should  see  a  dramatic  increase  in  the  number  of  CASE 
tools  users  in  all  phases  of  software  development;  not  just 
requirements  analysis. 

Research  Question  #2.  The  question  was:  For  what 
reasons  did  agencies  choose  to  use  CASE  tools  or  not  to  use 
CASE  tools? 
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The  data  reviewed  indicated  that  the  population  was 
still  uncertain  of  using  CASE  tools.  Many  organizations  did 
not  use  the  tools  because  of  the  expense  involved,  the  lack 
of  understanding  into  how  the  tools  worked  or  could  help, 
lack  of  training  of  personnel,  or  the  total  rejection  of  the 
CASE  tool  ideas.  Education  into  the  benefits  of  CASE  tools 
is  needed  throughout  the  Air  Force.  The  Software 
Engineering  Professional  Continuing  Education  is  a  leading 
effort  in  the  education  and  more  efforts  like  this  are 
needed  to  increase  the  CASE  tool  awareness  of  software 
developers . 

Research  Question  #3.  The  question  was;  What  was  the 
project  size  (lines  of  code.  Cost)?  Was  projec.  si^  a 
factor  in  deciding  whether  or  not  to  use  CASE  tools  on  the 
Project? 

The  data  analyzed  portrayed  that  most  software 
development  projects  are  of  a  significant  size.  This 
coupled  with  the  organizational  and  personal  CASE  tool  use 
levels  leads  one  to  conclude  that  the  size  of  a  project  was 
not  a  factor  in  determining  whether  or  not  to  use  CASE 
tools.  Personal  or  organizational  preferences  tend  to  be 
the  driving  factor  on  CASE  tool  use. 

Research  Question  #4.  The  question  was:  What  were  the 
initial  costs  of  using  the  tools?  Follow-on  or  recurring 
costs? 
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Not  knowing  the  data  was  the  significant  finding  in 
this  area.  Organizations  that  use  CASE  tools  do  not 
routinely  track  the  costs  of  using  the  tools  or  do  not  give 
the  information  out  to  everyone  in  the  organization.  A 
better  way  of  obtaining  the  data  for  future  research  efforts 
must  be  found.  Whether  the  budget  offices  are  surveyed  or 
the  data  is  requested  in  a  different  manner  is  up  to  future 
researchers . 

Research  Question  #5.  The  question  was:  What  cost 
savings  were  found  by  using  CASE  tools  over  not  using  CASE 
tools  (including  the  recovery  of  the  initial  costs)? 

The  conclusions  drawn  in  this  area  are  simple.  Not 
*  enough  data  is  being  tracked  to  make  any  assertions  one  way 
or  another  in  regards  to  costs.  With  the  tightening  of  the 
budget  expected  over  the  next  few  years,  organizations  need 
to  find  areas  in  which  they  can  track  costs  in  order  to 
reduce  them.  Tracking  of  the  costs  of  errors  and  the  phase 
of  development  in  which  they  occurred  will  enable  managers 
to  better  understand  where  problem  areas  are.  Knowing  where 
the  problems  are  is  the  first  step  in  reducing  the  costs  of 
errors  . 

Research  Question  #6.  The  question  was:  What  errors 
were  found?  When?  Did  using  CASE  tools  reduce  the  number  of 
errors? 

The  results  in  this  area  are  the  same  as  the  results 
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for  research  question  #5.  Not  enough  data  is  being  tracked 
to  ascertain  any  significant  findings  in  this  area.  The 
opinion  of  the  population  is  one  that  using  CASE  tools  will 
reduce  errors,  but  so  far  the  opinions  have  not  been  heeded 
or  acted  upon. 

Overall  Conclusions 

The  software  development  organizations  throughout  the 
Air  Force  are  just  getting  on  the  CASE  tool  bandwagon.  The 
effects  of  CASE  tools  on  projects  has  yet  to  be  realized. 
Efforts  are  underway  to  increase  the  use  and  awareness  of 
CASE  tools.  The  upcoming  release  of  the  ICASE  request  for 
proposal  is  a  giant  step  toward  getting  CASE  tools  into  the 
hands  of  the  developers.  These  integrated  tools  will  enable 
the  developers  to  engineer  projects  with  greater  ease  and 
accuracy.  These  factors  should  manifest  an  overall  savings 
in  time  and  money  needed  to  develop  and  maintain  software. 
The  Air  Force  Institute  of  Technology's  Software  Engineering 
Professional  Continuing  Education  program  provides 
developers  the  opportunity  to  learn  the  current  methods  and 
practices  of  the  ever-changing,  ever-growing  profession  of 
software  engineering.  By  keeping  up  with  the  changes  in  the 
software  world,  costs  can  be  saved  and  even  avoided. 
Recommendations  for  Future  Research 

An  enormous  amount  of  possible  research  endeavors  in 
the  area  of  CASE  tools  remain.  With  the  Air  Force  just 
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releasing  the  RFP  for  an  integrated  CASE  tool  set,  the  case 
study  possibilities  are  numerous.  Also,  there  are 
possibilities  for  a  more  detailed  analysis  of  the  Air  Force 
as  a  whole  using  descriptive  statistics. 

The  case  study  approach  has  two  immediate 
possibilities.  First,  review  of  a  completed  software 
project,  where  CASE  tools  were  used  in  a  part  of  the 
development,  can  be  undertaken.  This  study  could  provide 
insight  into  the  effectiveness  of  early  CASE  tools,  how  and 
where  they  were  used,  and  lessons  learned  by  the  developing 
organization.  Second,  tracking  a  new  system  being  developed 

using  the  new  ICASE  toolkit  is  a  long  range  project.  This 

% 

study  could  provide  feedback  to  the  practicality  of  the 
ICASE  product  and  its  implementation. 

In  completing  a  more  detailed  analysis  of  the  Air 
Force,  one  would  need  to  build  a  much  more  thorough  survey 
instrument  to  obtain  the  required  exacting  data.  By 
collecting  this  type  of  data,  more  precise  quantitative 
statistics  can  be  derived.  With  greater  precision,  one  can 
draw  more  specific  conclusions  about  the  Air  Force 
population  as  a  whole  and  as  various  subgroups. 

Summary 

The  thesis  was  undertaken  in  an  effort  to  try  and 
provide  insight  and  understanding  into  the  current  use  of 
CASE  tools  in  the  requirements  analysis  phase  of  software 
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development.  Expectations  were  ones  that  the  Air  Force  did 
not  have  great  efforts  into  this  area  of  tool  use,  that  data 
tracking  in  these  areas  was  lacking,  and  that  improvements 
were  needed  in  all  areas  related  to  CASE  tools.  The 
findings  of  the  research  supported  the  expectations  of  the 
author.  Much  work  still  remains  in  the  CASE  tool  arena.  In 
fact,  the  work  is  only  beginning.  CASE  tools  can  and  will 
make  tremendous  differences  in  future  software  development 
projects  or  efforts. 
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Appendix  A:  Initial  Survey  Package 


Survey  Approval  Request 

This  appendix  contains  the  information  sent  to  HQ  AFMPC 
to  request  approval  to  distribute  a  survey. 


Request  Form 

REQUEST  FOR  APPROVAL  OF  SURVEY 

AFIT/GSS/ENG/91D-7 
Capt  M.  B.  Key,  Jr. 


1.  TITLE  OF  PLANNED  SURVEY. 

Computer-Aided  Software  Engineering  (CASE)  Tool  Use 

Survey 

2.  NAME  AND  ADDRESS  OF  INDIVIDUAL  REQUESTING  APPROVAL. 

AFIT  Survey  Control  Officer 

AFIT/XP 

Capt  Frisco-Foy  (DSN  785-4219) 

Wright-Patterson  AFB  45433-6583 

3.  STATEMENT  OF  SURVEY  PROBLEM,  SURVEY  PURPOSE,  PREFERRED 

ADMINISTRATION  TIME  FRAME,  AND  JUSTIFICATION. 

a.  SURVEY  PROBLEM. 

Software  projects  have  become  enormous  and  extremely 
complex.  To  assist  the  future  software  engineer  in 
managing  this  growing  complexity,  today's  engineers 
are  creating  automated  tools  to  simplify  the 
software  management  and  development  process.  These 
products  are  called  Computer-Aided  Software 
Engineering  (CASE)  tools.  To  be  effective,  CASE 
tools  must  be  used  from  the  beginning  of  the 
software  development  process.  No  research  has  been 
conducted  to  determine  the  imparts  of  using  these 
tools  during  requirements  analysis  for  Air  Force 
software  development  projects. 


b.  SURVEY  PURPOSE. 

The  intent  of  this  questionnaire  is  to  gather 
sufficient  data  to  determine  whether  using  CASE 
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tools  during  requirements  analysis  reduces  errors  in 
the  overall  development  cycle  and,  ultimately,  save 
money . 

c.  PREFERRED  ADMINISTRATION  TIME  FRAME. 

As  soon  as  practical.  Survey  data  must  be  collected 
by  10  July  1990  for  meaningful  analysis. 


d.  JUSTIFICATION  FOR  SURVEY. 

The  information  collected  will  aid  in  the  analysis 
of  CASE  tools  use  during  requirements  analysis  in 
the  Air  Force.  This  research  effort  is  also  a 
requirement  of  the  Master  of  Science  thesis  at  the 
Air  Force  Institute  of  Technology. 

4.  HYPOTHESIS  TO  BE  TESTED  BY  THE  SURVEY. 

CASE  tool  use  during  requirements  analysis  correlates 
directly  to  a  reduction  of  errors  attributable  to  poor 
requirements  analysis  resulting  in  a  reduction  of 
deficiency  and/or  error  corrections  in  later  stages  of  a 
project . 

5.  POPULATION  TO  BE  SURVEYED. 

a.  DESCRIPTION  OF  POPULATION  TO  INCLUDE  MILITARY  AND 
CIVILIAN  EMPLOYMENT  STATUS  AND  BY  WHOM  EMPLOYED. 

Software  project  managers  and  programmers  (of  all 
ranks  and  grades)  at  all  CONUS  USAF  software 
development  organizations. 

b.  SIZE  OF  POPULATION. 

Approximately  20,000. 

6.  DESCRIPTION  AND  SIZE  OF  SAMPLE  SELECTED. 

a.  SAMPLE  DESCRIPTION. 

The  sample  will  consist  of  individuals  (both 
military  and  civilian)  who  work  directly  on  USAF 
software  development  projects. 

b.  SIZE  OF  SAMPLE  AND  IP  INDIVIDUAL  RESPONDEES  ARE 
KNOWN  OR  NOT  KNOWN. 

Sample  size  will  be  600.  The  names  of  the 
individual  respondents  are  unknown. 
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7.  METHOD  OF  SELECTION  OF  THE  SAMPLE. 
Simple  random  sample. 


8.  METHOD  OF  CONDUCTING  THE  SURVEY. 

Questionnaires  will  be  distributed  to  subjects  through 
their  parent  organization.  Completed  questionnaires 
will  be  individually  return  mailed. 

9.  DESCRIPTION  OF  THE  STATISTICAL  ANALYSIS  PLAN  OR  OTHER 
METHOD  OF  EVALUATION. 

Analysis  of  variances,  testing  of  means,  correlation  of 
populations,  and  basic  averages  and  percentages. 

10.  METHOD  OF  TABULATING  SURVEY  RESULTS. 

Computer  tabulation  of  answer  sheets  and  manual  coding 
of  write-in  responses  into  categories  and  histogram 
analysis  and  reporting. 

11.  USE  AND  DISPOSITION  OF  RESULTS. 

The  survey  results  will  be  basic  inputs  to  a  published 
Air  Force  Institute  of  Technology,  School  of  Systems 
and  Logistics,  student  thesis.  The  results  may  be 
released  to  the  public  and  the  Air  Force  Human  Resources 
Laboratory  in  compliance  with  AFR  12-30. 

12.  COMMAND  APPROVAL  CONTACT  POINT. 

Capt  Brad  Ayers 
Hq  AFCC/PGSS 
Scott  AFB  II 
DSN  576-3642 

13.  Copy  of  the  proposed  survey  questionnaire,  including 
Privacy  Act  Statement,  is  attached. 
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Survey  Cover  Letter 
LSG  (Capt  Key,  DSN  785-8989) 
CASE  Tool  Use  Survey 


USAF  Software  Managers  and  Developers 

1.  Please  take  the  time  to  complete  the  attached 
questionnaire . 

2.  The  survey  measures  Computer-Aided  Software  Engineering 
(CASE)  tool  use  during  the  requirements  analysis  activities 
of  software  development  in  USAF  software  development 
agencies.  The  data  gathered  will  become  part  of  an  AFIT 
research  project  and  may  influence  approaches  to 
requirements  analysis  in  future  software  projects.  Your 
individual  responses  will  be  combined  with  others  and  will 
not  be  attributed  to  you  personally. 

3.  Your  participation  is  completely  voluntary,  but  we 
certainly  appreciate  your  help.  For  further  information, 
contact  Lt  Col  Lawlis  at  DSN  785-6027  or  Capt  Key  at  DSN 
785-8989. 


JOHN  W.  SHISHOFF,  Lt  Col,  USAF 
Director,  Graduate  Programs 
School  of  Systems  and  Logistics 


4  Atch 

1.  Privacy  Act  Statement 

2.  Questionnaire 

3.  Computer  Answer  Sheet 

4.  Return  Envelope 
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Privacy  Act  Form 


PRIVACY  ACT  STATEMENT 


In  accordance  with  paragraph  8,  AFR  12-35,  the  following 
information  is  provided  as  required  by  the  Privacy  Act  of 
1974: 

a.  Authority 

(1)  10  U.S.C.  301,  Departmental  Regulations; 
and/ cr 

(2)  10  U.S.C.  8012,  Secretary  of  the  Air  Force, 
Powers,  Duties,  Delegation  by  Compensation; 
and/or 

(3)  EO  9397,  22  Nov  43,  Numbering  System  for 
Federal  Accounts  Relating  to  Individual 
Persons;  and/or 

(4)  DOD  Instruction  1100.13,  17  Apr  68,  Surveys 
of  Department  of  Defense  Personnel;  and/or 

(5)  AFR  30-23,  22  Sep  76,  Air  Force  Personnel 
Survey  Program. 

b.  Principal  Purposes.  The  survey  is  being  conducted 
to  collect  information  to  be  used  in  research  aimed 
at  illuminating  and  providing  inputs  to  the 
solution  of  problems  of  interest  to  the  Air  Force 
and/or  DOD. 

c.  Routine  Uses.  The  survey  data  will  be  converted  to 
information  for  use  in  research  of  management 
related  problems.  Results  of  the  research,  based 
on  the  data  provided,  will  be  included  in  written 
master's  theses  and  may  also  be  included  in 
published  articles,  reports,  or  texts. 

Distribution  of  the  results  of  the  research,  based 
on  the  survey  data,  whether  in  written  form  or 
presented  orally,  will  be  unlimited. 

d.  Participation  in  this  survey  is  entirely  voluntary. 

e.  No  adverse  action  of  any  kind  may  be  taken  against 
aiiy  individual  who  elects  not  to  participate  in  any 
or  all  of  this  survey. 


A-5 


Proposed  Survey  Instrument 


Computer-Aided  Software  Engineering  (CASE)  Tool  Use  Survey 


It  is  estimated  that  the  United  States  Air  Force  (USAF) 
encountered  $30  billion  in  software-related  costs  in  FY 
1990.  Of  this  amount,  software  development  and  maintenance 
accounted  for  about  80%  of  the  total  estimate.  Cheaper  and 
more  efficient  ways  of  working  with  software  must  be 
developed  and  used  in  order  to  reduce  the  skyrocketing 
development  and  maintenance  expenditures. 

CASE  tools  for  requirements  anal^-sis  are  an  important 
part  of  the  CASE  family.  CASE  tool  use  can  improve  the 
accuracy  and  traceability  of  requirements  throughout  the 
software  development  lifecycle.  A  proper  requirements 
analysis  can  save  thousands,  if  not  millions  of  dollars 
annua  1 1 y . 

This  survey  is  being  sent  to  all  CONUS  USAF  software 
development  organizations.  The  survey  is  also  being  given 
to  Professional  Continuing  Education  students  who  attend 
courses  at  the  Air  Force  Institute  of  Technology.  Please 
answer  the  questions  based  on  your  organization  and  the 
programs  that  you  are  responsible  for. 

An'  comments,  suggestions,  and  explanations  are 
welcomed  and  encouraged.  All  inputs  will  be  considered  in 
the  final  report.  Address  all  inquiries  to  Capt  Marvin  Key, 
AFIT/LSG,  Wright-Patterson  AFB  OH  45433  (DSN  785-8989). 


Survey  Completion 


This  survey  contains  20  questions.  For  questions  1-16, 
circle  the  appropriate  answer.  For  questions  17-20,  write 
your  answer  after  each  question.  Accompanying  the  survey, 
you  will  find  a  computer  answer  sheet.  Please  mark  your 
answers  on  the  computer  answer  sheet  after  completing  the 
survey . 
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1.  what  is  your  rank? 

a.  Enlisted  c.  04  to  06  e.  Below  GS-10  g.  GS/Od-lS  to  GS/GM-15 

b.  01  to  03  d.  07  to  10  f.  GS-10  to  GS-12  h.  Above  GS/GM-15 

2.  How  many  years  of  software  progranming  experience  do  you  have? 

a.  Less  than  1  year  c.  3  to  6  years  e.  More  than  10  years 

b.  1  to  3  years  d.  6  to  10  years 

3.  What  kind  of  systems  does  your  office  develop? 

a,  Ccmmand  and  control  c.  Avionics  e.  Other  embedded/ real -time  ( _ ) 

b.  Intelligence  d.  Mgmt  Information  f.  Other  (Please  specify  _ ) 

4.  How  many  systems  does  your  organization  have  in  development  at  the  present  time? 

a.  0  b.  1  to  2  c.  3  to  5  d.  6  to  9  e.  10  or  more 

5.  What  is  the  average  cost  of  a  system  under  development  in  your  organization? 

a.  $0  to  9,999  c.  $50,000  to  99,999  e.  $1,000,000  or  more 

b.  $10,000  to  49,999  d.  $100,000  to  999,999 

6.  What  is  the  average  software  project  size  you  develop  (in  lines  of  code)? 

a.  Less  than  500  c.  1,000  to  9,999  e.  More  than  100,000 

b.  500  to  999  d.  10,000  to  99,999 

7.  In  what  areas  (mark  all)  of  the  software  lifecycle  are  CASE  tools  used  in  your  office? 

a.  Requirements  analysis  c.  Coding  e.  Maintenance  g.  Do  not  know 

b.  Design  d.  Testing  f.  None 

S.  What  was  the  initial  cost  of  using  CASE  tools  (include  purchase  price,  any  special 
hardware  and  initial  training  costs)? 

a.  $0  to  999  c,  $10,000  to  49,999  e.  $100,000  or  more 

b.  $1,000  to  9,999  d.  $50,000  to  99,999  (Please  specify  _ ) 

9.  What  are  your  annual  recurring  costs? 

a.  $0  to  999  c.  $5,000  to  9,999  e.  $50,000  or  more 

b.  $1,000  to  4,999  d.  $10,000  to  49,999  (Pleaise  specify _ ) 

10.  Where  were  you  first  exposed  to  CASE  tools? 

a.  On  the  job  b.  In  school  c.  Magazines  or  journals  d.  Other  (Please  specify 

_ ) 

11.  How  long  has  your  organization  been  using  CASE  tools? 

a.  Less  than  1  year  b.  1  to  2  years  c.  3  to  4  years  d.  More  than  4  years 
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12.  On  your  projects  that  do/did  NOT  use  CASE  tools,  how  many  errors  per  thousand  lines 
of  code  (KLOC)  can  be  traced  back  to  problems  with  requirements  analysis  (please 
provide  as  accurate  a  number  as  possible)? 

a.  Less  than  10  errors/KLOC  c.  26  to  50  errors/KLOC  e.  More  than  100  errors/KLOC 

b.  10  to  25  errors/KLOC  d.  51  to  100  errors/KLOC  (Please  specify  _ ) 

13.  What  percentage  of  total  errors  did  requirements  definition  error  account  for  in  the 
projects  listed  above? 

a.  Less  than  5%  c.  10  to  25%  e.  More  than  50% 

b.  5  to  10%  d.  25  to  50%  (Please  specify _ ) 

14.  On  your  projects  that  do/did  use  CASE  tools.  How  many  errors  per  thousand  lines  of 
code  can  be  traced  back  to  problems  with  requirements  analysis  (please  provide  as 
accurate  a  number  as  possible  for  each  project)? 

a.  Less  than  10  errors/KLOC  c.  26  to  50  errors/KLOC  e.  More  than  100  errors/KLOC 

b.  10  to  25  errors/KLOC  d.  51  to  100  errors/KLOC  (Please  specify  _ ) 

15.  What  percentage  of  total  errors  did  requirements  definition  error  account  for  in  the 
projects  listed  above? 

a.  Less  than  5%  c.  10  to  25%  e.  More  than  50% 

b.  5  to  10%  d.  25  to  50%  (Please  specify _ ) 

16.  If  used,  why  were  CASE  tools  selected  to  be  used  for  requirements 
analysis? 


a.  Tools  made  task  easier 

b.  Analysis  more  accurate 

c.  Directed  to  use  tools 


d.  Done  as  a  test  case 

e.  Not  used 

f.  Other  (Please  specif y_ 


Respond  to  the  next  four  statements  using  the  following  scale: 


Strongly  disagree 

1 

Disagree 

r 

f 

Neither 

1 

1 

Agree 

1 

1 

Strongly  agree 

1 

1 

A 

B 

C 

D 

E 

,) 


17.  Errors  increased  after  using  CASE  tools. 

18.  Productivity  increased  when  using  CASE  tools. 

19.  By  using  CASE  tools,  overall  development  costs  increased. 

20.  Using  CASE  tools,  time  to  conplete  the  projects  increased. 
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endix  B:  Air  Force  Manpower  and  Personnel  Center  Response 


'.'n-’w  DPMVOS  *  IJbl 

Survey  Approval  (Your  Ltr,  1  May  1991) 


">  AFIT/XPX 

1.  Capt  Key's  survey,  “Computer-Aided  Software  Engineering 
(CASE)  Tool  Use,“  is  approved,  contingent  upon  the  changes  listed 
below.  The  Survey  Control  Number  (USAF  SCN)  91-32,  and  the 
expiration  date  of  30  September  1991  must  appear  on  the  cover  of 
the  survey.  Please  have  Capt  Key  forward  a  copy  of  the  corrected 
survey  to  this  office  prior  to  administration. 

2.  The  major  changes  apply  to  the  population  definition,  sample 
selection,  and  scope  of  the  survey.  The  defined  population  needs 
to  be  much  more  select  in  order  to  get  a  sample  of  personnel  who 
might  conceivably  Know  what  CASE  tools  are  and  who  may  have  used 
them.  The  sample  required  for  a  90%ll0  confidence  Interval,  the 
allowable  interval  for  thesis  students,  is  67  for  any  population 
over  6'J41.  Capt  Key  is  authorized  to  survey  150  people,  'iho 
scope  of  the  survey  as  currently  written,  i.e.,  from  an 
organizational  perspective,  would  probably  only  be  Known  by  very 
senior  managers.  Therefore,  most  of  the  rewording  of  the 
questions  is  to  help  the  individual  relate  their  own  experience 
to  CASE  tools.  Capt  Key  also  needs  to  have  very  specific 
instructions  to  the  offices  distributing  the  surveys  in  order  to 
get  a  meaningful  sample. 

3.  Changes  to  the  survey  Instrument  are: 

a.  In  the  cover  letter,  taKe  out  the  instruction  to  include 
name,  ranK,  etc.,  on  the  answer  sheet. 

b.  On  the  general  instruction  page,  taKe  out  the  first  two 
paragraphs.  The  survey  respondent  doesn't  require  this 
information  in  order  to  complete  the  survey.  Also,  the 
respondent  should  be  instructed  to  complete  the  survey  using  the 
answer  sheet.  To  complete  the  survey  and  then  transfer  responses 
to  the  answer  shoot  is  not  necess.iry. 

c.  On  question  1,  delete  option  'd'  because  we  do  not 
recommend  surveying  general  officers.  Also,  enlisted  personnel 
may  taKe  offense  at  only  having  one  response  option.  Recommend 
grade  group  (E1-E3,  E4-E6,  E7-E9)  options  be  used  for  enlisted 
personnel . 

d-  On  question  4,  change  stem  to  read,  "How  many  systems  are 
you  developing  at  the  present  time?" 

e.  On  question  5,  change  the  stem  to  read,  "What  is  the 
average  cost  of  systems  you  develop?" 
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f.  On  question  6,  change  the  stem  to  read,  "What  la  the 
average  software  project  size  you  develop?" 

g.  After  question  6,  add  two  questions: 

(1.)  Does  your  organization  use  CASE  tools?  A.  Yes  B. 
No 

(2.)  Do  you  use  CASE  tools?  A.  Yes  B.  No 

h.  On  question  7,  change  the  stem  to  read,  "In  what  areac  of 
the  software  lifecycle  do  you  use  CASE  tools?" 

i.  On  questions  8  and  9,  add  response  options  for  "I  don't 
know. " 

j.  On  question  11,  change  stem  to  read,  "How  long  have  you 
been  using  CASE  tools?" 

k.  On  questions  12  and  14,  insert  the  word  "your"  prior  to 
the  word  "projects." 

l.  At  the  end  of  the  survey,  tell  the  respondents  how  to 
return  the  survey. 

4.  If  Capt  Key  has  any  questions  about  this  review  or  the 
required  changes,  he  may  contact  Capt  Burgess  at  DSN  487-5680. 
Again,  please  have  Capt  Key  send  us  a  copy  of  the  final  survey 
prior  to  .administration. 

CHARLES  H.  HAM  I LT I  ON)  GM-13 
Chief,  Personnel  Survey  Branch 
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After  reviewing  the  comments  and  suggests  from  AFMPC/DPMYOS , 


changes  were  made  to  the  original  survey  instrument, 
final  instrument  is  included  in  Appendix  C. 


The 
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Appendix  C:  Final  Survey  Package 


Cover  Letter 


LSG  (Capt  Key,  DSN  785-8989) 
CASE  Tool  Use  Survey 


USAF  Software  Managers  and  Developers 

1.  Please  take  the  time  to  complete  the  attached 
questionnaire . 

2.  Persons  completing  the  questionnaire  should  be  in  a 
position  of  managing  one  or  more  software  development 
projects,  managing  a  subsystem  of  a  larger  project,  or 
developing  specifications  and  code.  The  project  can  be  an 
in-house  development  or  a  contracted  effort.  All 
development  efforts  will  be  considered.  Also,  all 
experience  levels  will  be  covered  by  the  survey  instrument, 

3.  The  survey  measures  Computer-Aided  Software  Engineering 
(CASE)  tool  use  during  the  requirements  analysis  activities 
of  soHware  development  in  USAF  software  development 
agencies.  The  data  gathered  will  become  part  of  an  ’.FIT 
research  project  and  may  influence  approaches  to 
requirements  analysis  in  future  software  projects.  Your 
individual  responses  will  be  combined  with  others  and  will 
not  be  attributed  to  you  personally. 

4.  Your  participation  is  completely  voluntary,  but  we 
certainly  appreciate  your  help.  For  further  information, 
contact  Lt  Col  Lawlis  at  DSN  785-6027  or  Capt  Key  at  DSN 
785-8989, 


JOHN  W,  SHISHOFF,  Lt  Col,  USAF 
Director,  Graduate  Programs 
School  of  Systems  and  Logistics 


4  Atch 

1.  Privacy  Act  Statement 

2.  Questionnaire 

3.  Computer  Answer  Sheet 

4.  Return  Envelope 
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Privacy  Act  Statement 


PRIVACY  ACT  STATEMENT 


In  accordance  with  paragraph  8,  APR  12-35,  the  following 
information  is  provided  as  required  by  the  Privacy  Act  of 
1974: 

a.  Authority 

(1)  10  U.S.C.  301,  Departmental  Regulations; 
and/or 

(2)  10  U.S.C.  8012,  Secretary  of  the  Air  Force, 
Powers,  Duties,  Delegation  by  Compensation; 
and/or 

(3)  EO  9397,  22  Nov  43,  Numbering  System  for 
Federal  Accounts  Relating  to  Individual 
Persons;  and/or 

(4)  DOD  Instruction  1100.13,  17  Apr  68,  Surveys 
of  Department  of  Defense  Personnel;  and/or 

(5)  AFR  30-23,  22  Sep  76,  Air  Force  Personnel 
Survey  Program. 

b.  Principal  Purposes.  The  survey  is  being  conducted 
to  collect  information  to  be  used  in  research  aimed 
at  illuminating  and  providing  inputs  to  the 
solution  of  problems  of  interest  to  the  Air  Force 
and/or  DOD. 

c.  Routine  Uses.  The  survey  data  will  be  converted  to 
information  for  use  in  research  of  management 
related  problems.  Results  of  the  research,  based 
on  the  data  provided,  will  be  included  in  written 
master's  theses  and  may  also  be  included  in 
published  articles,  reports,  or  texts. 

Distribution  of  the  results  of  the  research,  based 
on  the  survey  data,  whether  in  written  form  or 
presented  orally,  will  be  unlimited. 

d.  Participation  in  this  survey  is  entirely  voluntary. 

e.  No  adverse  action  of  any  kind  may  be  taken  against 
any  individual  who  elects  not  to  participate  in  any 
or  all  of  this  survey. 
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Final  Survey  Instrument 


Computer-Aided  Software  Engineering  (CASE)  Tool  Use  Survey 


This  survey  is  being  sent  to  all  CONUS  USAF  software 
development  organizations.  The  survey  is  also  being  given 
to  Professional  Continuing  Education  students  who  attend 
courses  at  the  Air  Force  Institute  of  Technology.  Please 
answer  the  questions  based  on  your  organization  and  the 
programs  that  you  are  responsible  for. 

Any  comments,  suggestions,  and  explanations  are 
welcomed  and  encouraged.  All  inputs  will  be  considered  in 
the  final  report.  Address  all  inquiries  to  Capt  Marvin  Key, 
AFIT/LSG,  Wright-Patterson  AFB  OH  45433  (DSN  785-8989). 


Survey  Completion 


This  survey  contains  22  questions.  For  questions  1-18, 
circle  the  appropriate  answer.  For  questions  19-22,  write 
your  answer  for  each  question  in  the  blank  preceding  each 
question.  Accompanying  the  survey,  you  will  find  a  computer 
answer  sheet.  Please  write  your  computer  answer  sheet 
number  in  the  space  provided  after  the  last  question  (the 
number  can  be  found  on  the  lower  right  hand  corner  of  the 
front  page  of  the  answer  sheet).  Then,  using  a  #2  pencil, 
please  mark  your  answers  on  the  computer  answer  sheet 

If  you  have  any  comments,  please  include  them  in  the  space 
provided  after  the  Computer  Answer  Sheet  number  on  page  4  of 
the  questionnaire. 
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1.  What  is  your  rarik? 

a.  El  to  E3  d.  01  to  03  g.  GS-10  to  GS-12 

b.  E4  to  E6  e.  04  to  06  h.  GS/C^-13  to  GS/GM-15 

c.  E7  to  E9  f.  Below  GS-10  i.  Above  GS/(34-15 

2.  How  many  years  of  software  progranming  experience  do  you  have? 

a.  Less  than  1  year  c.  3  to  6  years  e.  More  than  10  years 

b.  1  to  3  years  d.  6  to  10  years 

3.  What  kind  of  systems  does  your  office  develop? 

a.  Conmand  and  control  c.  Avionics  e.  Other  embedded/ real -time  ( _ ) 

b.  Intelligence  d.  Mgmt  Information  f.  Other  (Please  specify  _ ) 

4.  How  many  systems  are  you  developing  at  the  present  time? 

a.  0  b.  1  to  2  c.  3  to  5  d.  6  to  9  e.  10  or  more 

5.  What  is  the  average  cost  of  systems  you  develop? 

a.  $0  to  9,999  c.  $50,000  to  99,999  e.  $1,000,000  or  more 

b.  $10,000  to  49,999  d.  $100,000  to  999,999 

6.  What  is  the  average  software  project  size  you  develop  (in  lines  of  code)? 

a.  Less  than  500  c.  1,000  to  9,999  e.  More  than  100,000 

b.  500  to  999  d.  10,000  to  99,999 

7.  Does  your  organization  use  CASE  tools? 

a.  Yes  b.  No 

8.  Do  you  use  CASE  tools? 

a.  Yes  b.  No 

9.  In  what  are:as  of  the  software  lifecycle  do  you  use  CASE  tools  (mark  all)? 

a.  Requirements  analysis  c.  Coding  e.  Maintenance  g.  Do  not  know 

b.  Design  d.  Testing  f.  None 

10.  What  was  the  initial  cost  of  using  CASE  tools  (include  purchase  price,  any  special 
hardware  and  initial  training  costs)? 


a.  $0  to  999 

d.  $50,000  to  99,999 

b.  $1,000  to  9,999 

e.  $100,000  or  more  (Pleaise  specify 

) 

c.  $10,000  to  49,999 

f .  Do  not  know 
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11.  What  are  your  annual  recurring  costs? 

a.  $0  to  999  d.  $10,000  to  49,999 

b.  $1,000  to  4,999  e.  $50,000  or  more  (Please  specify _ ) 

c.  $5,000  to  9,999  f.  Do  not  know 

12.  Where  were  you  first  exposed  to  CASE  tools? 

a.  On  the  job  b.  In  school  c.  Magazines  or  journals  d.  Other  (Please  specify 

_ ) 

13.  How  long  have  you  been  using  CASE  tools? 

a.  Less  than  1  year  b.  1  to  2  years  c.  3  to  4  years  d.  More  than  4  years 

14.  On  your  projects  that  do/ did  NOT  use  CASE  tools,  how  many  errors  per  thousand  lines 
of  code  (KLOC)  can  be  traced  back  to  problems  with  requirements  analysis  (please 
provide  as  accurate  a  number  eis  possible)? 

a.  Less  than  10  errors/KLOC  c.  26  to  50  errors/KLOC  e.  More  than  100  errors/KLOC 

b.  10  to  25  errors/KLOC  d.  51  to  100  errors/KLOC  (Please  specify _ ) 

15.  What  percentage  of  total  errors  did  requirements  definition  error  account  for  in  the 
projects  listed  above? 

a.  Less  than  5%  c.  10  to  25%  e.  More  than  50% 

b.  5  to  10%  d.  25  to  50%  (Please  specify _ ) 

16.  On  your  projects  that  do/did  use  CASE  tools.  How  many  errors  per  thousand  lines  of 
code  can  be  traced  back  to  problems  with  requirements  analysis  (please  provide  as 
accurate  a  number  as  possible  for  each  project)? 

a.  Less  than  10  errors/KLOC  c.  26  to  50  errors/KLOC  e.  More  than  100  errors/KLOC 

b.  10  to  25  errors/KLOC  d.  51  to  100  errors/KLOC  (Please  specify  _ ) 

17.  What  percentage  of  total  errors  did  requirements  definition  error  account  for  in  the 
projects  listed  above? 

a.  Less  than  5%  c.  10  to  25%  e.  More  than  50% 

b.  5  to  10%  d.  25  to  50%  (Please  specify _ ) 

18.  If  used,  why  were  CASE  tools  selected  to  be  used  for  requirements 
analysis? 

a.  Tools  made  task  easier  d.  Done  as  a  test  case 

b.  Analysis  more  accurate  e.  Not  used 

c.  Directed  to  use  tools  f.  Other  (Please  specify _ ) 
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Respond  to  the  next  four  statements  using  the  following  scale: 


Strongly  disagree 

t 

Disagree 

1 

» 

Neither 

1 

1 

Agree 

1 

1 

Strongly  agree 

1 

1 

A 

B 

C 

D 

E 

_  19.  Errors  increased  after  using  CASE  tools. 

_  20.  Productivity  increased  when  using  CASE  tools. 

_  21.  By  using  CASE  tools,  overall  development  costs  increased. 

_  22.  Using  CASE  tools,  time  to  corrplete  the  projects  increased. 

Computer  Answer  Sheet  Number  _ _ 

COMMENTS  (Continue  on  the  next  page  if  necessary) 
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RETURN  INSTRUCTIONS 

Once  you  have  filled  in 
and  answer  sheet  in  the 

Thank  you  for  your  time 


the  computer  answer  sheet,  insert  the  questionnaire 
return  envelope  and  place  the  envelope  in  the  mail. 

and  assistance. 
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Appendix  D  -  Control  Group  Raw  Data 


The  following  raw  data  was  obtained  from 
answer  sheets  filled  out  when  the  surveys  were 
The  data  contains  91  cases. 

DBCBEDBBFAAB 
GA  BDABBFFFCA 
DAFAAABBFAF  A  E 

GCFEE 

HBCBECABEFFDA  D 

GCCCDEBBF  FC  AA 
GAEA  BBFF 

GDEADDAB  FFAADDCC  BDCB 
GDDBECABGFFB  ABABECCCC 
GDEBEDBBF  AA  BDCB 

HCACEEBBFDFDD  AAEDD 

GACEDEABFFEAACCBDBBDBB 
HE  CEDBBG  C 
GBACEEBBFAFC  E 

GBCBBEABEFFCA  ADCC 

FBEBEEAB  FFAA  CBDDD 

GDDBEEAB  FFABD  CCDCC 

HCEEEEAB  FFD  ACCCC 

GCFBEDBB  F  C  AA  AE 
HE  AAABBFAAA  E 


the  computer 
comp  1 et  ed . 
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GD  BEDBB  F  C  AE 


HCCBEEBBGFFC 
DCAEDEBBFFFCA 
DCFA  ABBG  FD 
HEDEDEBBFFFC  E 

GBFAAABBGAADA 
GCEBEEBBFFFDABA  ADBB 
HA  BAABBGFAAAAA 
FD  ADCBBFFFBA  BDBB 

GCDB  AB  DFC  C 

DC  BEEAA  DFAC  ADCB 

DBEBDCBBGFFD 
HC  BEDAA  ECCA  E  D 

DAACE  BBGFFD 
DBABEDBBFAADAAD  CCCC 
GD  C  DBBFFFC 
GCEC  DBBGFFC 
GAFA  BB  FF 
GACCEEAABFFAC 
HDFCCDBBFFF  D 
DBACDCAA  CBBB  DADAC 

DCACDCAA  CBAB  DBCBB 

DCAA  AA  CAABBCBCEBCBB 
HCDECDBBFFFC 
DABBAABBFAAA  E 
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DCACECAAGFFAACC 


HECEEDAA  FFAD  BDCB 

GDDECEABGDAAB  E 
GDDECDAABBAACBCAAEBDBD 
HEF 

DBDECCABGFFABCD  CDBDBB 
GCACEDABBFFAA 
DCABDCBBFA\C  AA  E 
GDFEBBBBF  C  BA 
DCCCEEAB: D  AAEDDDBCDBC 
HECEEE\B  FFCD 
GBECEEBBF  AAA6  CCCC 
GAFBEDBBF 

GEDACEBBFAABAEA  ECCCC 
DCABEEAAAFFBO  ABDBB 
DDCAAABBFAAAA  E 
DCCEEEAB  FFB 
GBFCED  GFFB  FCCCC 

DDCDDD  F  AAAA 
E 

DBFAAA  G 

DA  E  D  G 

GEABBCBBFAACA 

DDAEEE  G  B 

DAEEECABGFFDDCBCCCC'^DD 


DC  EEEABGFFBBDC  D  DDDC 


DA  EEEA  EBCB  ACBB 

GCFAAEBBFAABAAAAAE 
DCCBEE  GFFB 
GDEAEDBBFFF 

HE  EEEAA  CC  A  ABDB 

GACBE  AA  FF 
GDBEEDBBFAAB  E 

GBF  F 

GD  B  B  F  AA  ECCCC 

GEEA  B  F  AA 

HEFA  B  FAADABD  E 

DAEBEE  G  B  E 

HDF  G  C 

EDF 

DCAACCABCFFCA 
GCEBEEBBEDA  A  ABDBB 

GD  BCCABGFFCA  D 

DBFBDEABFFEBA 
EEAEDDABFFDDABDADECCCC 


DCFBEABBFAFBAAA 


Appendix  E  -  Frequency  Distribution  Charts  of  Control  Groui 

Responses 

Introduction 

The  following  frequency  distribution  charts  represent 

the  answers  provided  by  the  control  group.  Of  the  22 

« 

questions,  questions  3,  9,  and  18  were  not  graphed  using  a 
frequency  distribution  because  respondents  could  provide 
multiple  answers  to  these  questions.  Tables  of  the  answers 
to  these  three  questions  can  be  found  in  Tables  4-1  through 
4-3  in  Chapter  IV. 

Frequency  Distribution  Charts 
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Figure  E-1.  Rank  and  Grade  of  Control  Group 
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Figure  E-2.  Experience  Levels  of  Control  Group 


Figure  E-3.  Number  of  Systems  in  Development  by  Control 


Group  Organizations 
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Figure  E-4.  Cost  of  Development  Systems  in  Control 

Group  Organizations 


Figure  E-5.  Average  Lines  of  Code  in  Development 
Systems  in  Control  Group  Organizations 
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Figure  E-6.  Determining  Number  of  Control  Group 
Organizations  that  Use  CASE  Tools 


Figure  E-7.  Determining  Number  of  Control  Group 
Respondents  that  Use  CASE  Tools 
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Figure  E-8.  Determining  Average  Initial  Costs  of  Using 
CASE  Tools  in  Control  Group  Organizations 


Figure  E-9.  Determining  Average  Recurring  Costs  of  CASE 


Tool  Use  in  Control  Group  Organizations 
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Figure  E-10.  Determining  First  Exposure  to  CASE  Tools 


in  Control  Group 


Figure  E-11.  Determining  CASE  Tools  Experience  Level  of 


Control  Group  Respondents 
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Errors  Uhile  Not  Using  CASE  Tools 


Figure  E-12.  Requirements  Analysis  Errors/KLOC  of 
Control  Group  Organizations  Not  Using  CASE  Tools 


Figure  E-13.  Percentage  of  Total  Errors  Credited  to 
Requirements  Analysis  in  Control  Group  Organizations  Not 

Using  CASE  Tools 


Errors/KLOC  Uhile  Using  CMC  Tools 


Figure  E-14.  Requirements  Analysis  Errors/KLOC  of 
Control  Group  Organizations  Using  CASE  Tools 
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Figure  E-15.  Percentage  of  Total  Errors  Credited  to 
Requirements  Analysis  in  Control  Group  Organizations 

Using  CASE  Tools 
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Figure  E-16.  Rating  Whether  Using  CASE  Tools  Increased 
Error  Rates  in  Control  Group 
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Figure  E-17.  Rating  Whether  Using  CASE  Tools  Increased 
Productivity  in  Control  Group 
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Figure  E-18.  Rating  Whether  Using  CASE  Tools  Increased 


Development  Costs  in  Control  Group 


Figure  E-19,  Rating  Whether  Using  CASE  Tools  Increased 


Time  to  Complete  Projects  in  Control  Group 
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The  following  raw  data  was  obtained  from  the  computer 
answer  sheets  filled  out  when  the  surveys  were  completed. 
The  data  contains  67  cases. 

DBDBCDAB  FFA 
DADAAAABCBBBAB  AD  BDCB 
DE  AEEBBFAAAA  ECCCC 
DBCAEDAAECCDA  DBCCC 
DB  DEEAB  FFAA  CBDCB 
DDBADCBBFFFBAC  E 
DAABACABFFFBABCACECDDD 
DEAAACBBFAABAAAAAECCCC 
DBFAAAAA  AABB  ACECA 
DBFBABBBFFFBACB  E 
DCABDDBBF  BADD  E 
DEDBAEAA  BABBBDCDCBCCD 
DCEBCDBBF  FBABB  F 
DDABEEAA  CFAAEEDE  AEBB 
ECBCEEBBBFFFBA  E 
EBCBEEAA  EEADCDACBAEBB 
EBABDDAA  CACB  ECCDB 
BBFBCCAA  FFABBBAAABCCC 
DDAAEDABGFFBAAAAAECCBB 
GDACBCBBF  D  AB 
CEADACBBFFFC  AAAAECCCC 
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DDDCDDABBFFBB  C  CEBDBB 


DEBACCABFFFCAAAABECCCC 

DEBCCCBBFFFCAAAAAE 

DCCAACBBFFF 

DCABDEAA  FFBB  ABCCC 

HEEAEEBB  A  BDBA 

DCBBBDBBF  B  BB  E 
GA  BBBAAEFAACBCAAD 
HECBEEAA  FFBDDDBC  BDBB 
DCABCDB3FFFBACCCCCBDDD 
GCBBEDBBFAAABDDDDAADBC 
BDDCDEBBFBACAAAAAECCCC 
GCEBEDBBFFFAABB  AEBB 
CC  CEEAB  FFAC  C  C 
IDDAB  CAA  BABB  BACCC 
ICABABAA  BAABAAAAECDCB 
GAFBCCBBFAABA  ECCCC 

HECBDDAAEDFBA  D  A 
GEFAAABBGAFDAABAAECCCC 
DDEBDCBBFAA  AA  AE 
HECCEDAA  EFCD  CECC 

BADBCDBB  FFA  AAAA  A 
BADBDDBBGFDAABBBBECCCC 
GD  BCDBBFABCABEBEECDDC 
GBDBEEBBF  D  DD  E 


HEDBDDAA  CCDADC  DBDCC 
DDDCCCAA  FAABBBAA  ADDB 
DCABCDABFFFABDC  CECCCC 
DADBBDAB  DDAABAAAABCDD 
GCCBBBABFFCBA 
GEBCDDAA  FFCAAAAAEBDBB 
DDBCACBBFFFBAAAAAEECBB 
DDBCDDABFFFCA 
lEFBBDCBBFFFCABBCECCCC 
DDDAAABBF  E 

CADBBDBBFADC  ECCCC 

DBDCEEAB  AEAC  AADCB 
DEBBDDAA  FFADCOAC  BEAB 
HEDBDDAA  CCCBCBBBEBDBB 
DCABEEBBFFFB  CCCC 

DBCBFEBBF  AA  E 
DAADEDABAFEAACCBBDBCBB 
DCABCDABFFFABDC  CECCCC 
BADBDDBBGFDAABBBBECCCC 
DDAAEDABGFFBAAAAAECCBB 
DEAAACBBFAABAAAAAECCCC 
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Appendix  G 


Frequency  Distribution  Charts  of  Sample  Grou 


Responses 

Introduction 

The  following  frequency  distribution  charts  represent 
the  answers  provided  by  the  sample  group.  Of  the  22 
questions,  questions  3,  9,  and  18  were  not  graphed  using  a 
frequency  distribution  because  respondents  could  provide 
multiple  answers  to  these  questions.  Tables  of  the  answers 
to  these  three  questions  can  be  found  in  Tables  4-4  through 
4-6  in  Chapter  IV. 

Fre^,uency  Distribution  Charts 
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Figure  G-1.  Rank  and  Grade  of  Sample  Group 
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Figure  G-2.  Experience  Levels  of  Sample  Group 


Figure  G-3.  Number  of  Systems  in  Development  by  Sample 


Group  Organizations 
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Figure  G-4.  Cost  of  Development  Systems  in  Sample  Group 

Organizations 
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Figure  G-5.  Average  Lines  of  Code  in  Development 
Systems  in  Sample  Group  Organizations 
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Figure  G-6.  Determining  Number  of  Sample  Group 
Organizations  that  Use  CASE  Tools 


Figure  G-7 .  Determining  Number  of  Sample  Group 
Respondents  that  Use  CASE  Tools 
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Figure  G-8.  Determining  Average  Initial  Costs  of  Using 
CASE  Tools  in  Sample  Group  Organizations 
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Figure  G-10.  Determining  First  Exposure  to  CASE  Tools 

in  Sample  Group 


Figure  G-11.  Determining  CASE  Tool  Experience  Level  of 

Sample  Group  Respondents 
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Figure  G-13.  Percentage  of  Total  Errors  Credited  to 
Requirements  Analysis  in  Sample  Group  Organizations  Not 

Using  CASE  Tools 
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Figure  G-14.  Requirements  Analysis  Errors/KLOC  of 
Sample  Group  Organizations  Using  CASE  Tools 
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Figure  G-15.  Percentage  of  Total  Errors  Credited  to 
Requirements  Analysis  in  Sample  Group  Organizations 

Using  CASE  Tools 
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Figure  G-16.  Rating  Whether  Using  CASE  Tools  Increased 
Errors  Rates  in  Sample  Group 
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Figure  G-17.  Rating  Whether  Using  CASE  Tools  Increased 
Productivity  in  Sample  Group 
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Figure  G-18.  Rating  Whether  Using  CASE  Tools  Increased 
Development  Costs  in  Sample  Group 
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Figure  G-19.  Rating  Whether  Using  CASE  Tools  Increased 
Time  to  Complete  Projects  in  Sample  Group 
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Appendix  H  -  Frequency  Distribution  Charts  of  All  Responses 

in  Combination 

Introduction 

The  following  frequency  distribution  charts  represent 
the  answers  provided  by  both  the  control  group  and  the 
sample  group.  Of  the  22  questions,  questions  3,  9,  and  18 
were  not  graphed  using  a  frequency  distribution  because 
respondents  could  provide  multiple  answers  to  these 
questions.  Tables  of  the  answers  to  these  three  questions 
can  be  found  in  Tables  4-7  through  4-9  in  Chapter  IV. 

Frequency  Distribution  Charts 
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Figure  H-1.  Rank  and  Grade  of  All  Respondents 
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Figure  H-2.  Experience  Levels  of  All  Respondents 


Figure  H-3.  Number  of  Systems  in  Development  by  All 


Surveyed  Organizations 


H-2 


70-1 


111 


Figure  H-4.  Cost  of  Development  Systems  in  All  Surveyed 

Groups 
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Figure  H-6.  Determining  Number  of  Surveyed 
Organizations  that  Use  Case  Tools 


Figure  H-7.  Determining  Number  of  Total  Respondents 

that  Use  CASE  Tools 
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Figure  H-9.  Determining  Average  Recurring  Costs  of  CASE 
Tool  Use  in  All  Surveyed  Organizations 
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Figure  H-10.  Determining  First  Exposure  to  CASE  Tools 

In  All  Respondents 


Figure  H-11.  Determining  CASE  Tool  Experience  Level  of 


All  Respondents 


H-6 


Errors  Uhilr  Mol  Usjnq  CASE  Tools 

Figure  H-12.  Requirements  Analysis  Errors/KLOC  of  All 
Surveyed  Organizations  Not  Using  CASE  Tools 


Figure  H-13.  Percentage  of  Total  Errors  Credited  to 
Requirements  Analysis  in  All  Surveyed  Organizations  Not 

Using  CASE  Tools 


Figure  H-14.  Requirements  Analysis  Errors/KLOC  of  All 
Surveyed  Organizations  Using  CASE  Tools 


Figure  H-15.  Percentage  of  Total  Errors  Credited  to 


Requirements  Analysis  in  All  Surveyed  Organizations 


Using  CASE  Tools 
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Figure  H-16.  Rating  Whether  Using  CASE  Tools  Increased 
Error  Rates  in  All  Surveyed  Organizations 
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Figure  H-17.  Rating  Whether  Using  CASE  Tools  Increased 
Productivity  in  All  Surveyed  Organizations 
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Figure  H-18.  Rating  Whether  Using  CASE  Tools  Increased 
Development  Costs  in  All  Surveyed  Organizations 


Figure  H-19.  Rating  Whether  Using  CASE  Tools  Increased 


Time  to  Complete  Projects  in  All  Surveyed  Organizations 
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Appendix  I  -  SAS  Analysis  of  Data 

Introduction 

The  following  tables  are  X  x  2  contingency  tables.  The 
X  represents  the  number  of  possible  responses  to  the 
particular  survey  plus  one.  The  one  represents  the 
possibility  of  having  a  response  answer.  The  2  in  the 
formula  represents  the  number  of  groups  surveyed;  the 
control  group  and  the  sample  group.  Also  included  in  these 
tables  are  the  Chi-Square  calculations  for  each  factor.  The 
description  and  explanation  of  these  tables,  factors,  and 
calculations  are  explained  and  analyzed  in  Chapter  IV. 

Tables  and  Chi-Sauare  Calculation^ 

Beginning  on  the  next  page,  nineteen  contingency  tables 
(including  Chi-Square  cal cul ations)  are  displayed. 

Questions  3,  9,  and  18  were  not  included  in  the  tables  or 
Chi-Square  calculations  because  of  the  respondents'  ability 
to  provide  multiple  answers  to  these  specific  questions. 
These  tables  and  calculations  were  generated  using  the  SAS 
Statistical  program.  19  individual  programs  were  created 
using  the  data  compiled  in  Appendices  E,  G,  and  H  as  the 
inputs.  Copies  of  these  programs  are  not  included  in  this 
document,  but  can  be  provided  upon  request. 
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Table  I-l.  Rank/Grade  of  Respondents 

TABLE  OF  GRADS  BY  GROUP 
GRADE  GROUP 

Fcaquancp I 
P  «  r  c  9  n  t  1 
Row  Pet  I 


Col  Pet 
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RAHK/GRADS  OP  REBPOHDERTS 

BTATISTICa  FOR  TABLE  OF  GRADS  BY  CROUP 


Statistic 

D  F 

V  a 

1  u  a 

P  c  o  b 
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NARHING:  67%  of  thw  calls  have  aapactad  counts  lass 

than  5.  Chi-Squara  may  not  bo  a  valid  last. 
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Table  1-2.  Years  of  Experience  of  Respondents 

TABLE  or  YEARS  BY  GROUP 

YEARS  GROUP 

Praquaacr I 
P  •  c  c  •  n  t  I 
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Table  1-3. 

Number 

of  Systems 

Under 

Development 
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1 

1 

2  3  . 

0  8 

1  0  . 

0  0 

1 

1 

-  -  - 

4  -  - - - 

,  _  - 

♦ 

3  -  5  S  Y  8  ! 

1  6 

1  2 

J 

2  8 

1 

t 

1  0  . 

1  3 

I  7  , 

5  9 

1 

17.72 

( 

5  7  . 

1  4 

i  4  2. 

8  6 

1 

1 

1 

1 

1  7  . 

5  8 

i  17. 

9  1 

1 

-  -  - - - - -f 

.  -  - 

4  -  -  - - 

-  -  - 

•  ¥ 

6  -  9  8  Y  8  ; 

1 

3 

4 

1 

1 

0  . 

6  3 

;  1 . 

9  0 

1 

1 

2.53 

i 

2  5  . 

0  0 

1  7  5. 

0  0 

1 

1 

1  . 

1  0 

1  4  . 

4  8 

t 

---------- 

4  - 

•  ♦ 

N  R  ! 

5 

0 

1 

5 

3  . 

1  6 

1  0  . 

0  0 

3.16 

1 

1 

10  0. 

0  0 

!  0  . 

0  0 

1 

; 

5  . 

4  9 

;  0  . 

0  0 

1 

1 

-  _  - 

4  -  - - 

_  -  . 

•  ¥ 

Total 

9  1 

6  7 

15  8 

5  7  . 

5  9 

4  2  . 

.  4  1 

10  0. 

0  0 

NUMBER  OP  SYSTEMS  UNDER  DEVELOPMENT 


STATISTICS  FOR  TABLE  OP  NUMSYS  BY  GROUP 


Statistic 

DP 

V 

a  1  u  e 

P  t  o  b 

Chi-Square 

5 

2  5 

.16  1 

0.000 

Likelihood  Ratio 

Chi 

-Square 

5 

3  4 

.659 

0.000 

Mantel  -Haenszel 

Chi- 

Square 

1 

3 

.587 

0.058 

Pisher's  Esact  T 

e  s  t 

( 2 -Tai 1 ) 

7  .  1  2  E  -  0  6 

Phi  Coetticient 

0 

.399 

Continqencr  Coot 

t  i  c  i 

•  at 

0 

.371 

Cramer's  V 

0 

.399 

Sampl*  Sis*  iS8 

NARRING;  33%  ot  th«  c«lls  hav*  aspscted  counts  less 
than  S.  Chi-Squace  mar  not  be  a  valid  test. 
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Table  1-4.  Cost  of  Systems  Under  Development 

TABLB  or  COST  BY  CROUP 
COST  GROUP 


Prvquancy 

Bspactad 

1 

P  •  I  c  •  n  t 

Row  Pet 

Col  Pet 

C  0  N  T 

1 

1 

SAMPLE 

1 

1 

Total 

f 

f 

1  M 

4  2 

1 

1 

1  9 

4 

1 

6  1 

3  5,133 

1 

1 

2  5.8  6  7 

1 

1 

2  6.58 

1 

1 

12.03 

« 

1 

3  8 

.  6  1 

6  8.85 

t 

1 

3  1.15 

4 

1 

4  6.15 

1 

t 

2  8.36 

4 

1 

♦ 

f 

4- 

4  9  .  9  K 

3 

1 

1 

8 

1 

1 

1  1 

6.3  3  5  4 

1 

t 

4.6  6  4  6 

4 

1 

1.90 

t 

5.06 

1 

1 

6 

.  9  6 

2  7.27 

1 

1 

7  2.73 

1 

1 

3.30 

1 

1 

11.94 

1 

( 

f 

f 

9  .  9  K 

8 

t 

1 

1  3 

1 

1 

2  1 

12.095 

1 

1 

8.9051 

1 

1 

5.06 

1 

1 

8.23 

1 

1 

1  3 

.  2  9 

3  8.10 

( 

1 

6  1.90 

1 

1 

8.79 

1 

1 

19.40 

1 

1 

f 

f 

f 

9  9  .  9  K 

7 

t 

• 

1  2 

1 

1 

1  9 

10.943 

« 

1 

8.057 

t 

« 

4.43 

1 

1 

7.59 

1 

1 

1  2 

.  0  3 

3  6.84 

1 

1 

6  3.16 

1 

1 

7.69 

1 

1 

17.91 

» 

t 

f 

f 

♦ 

9  9  9.  9  K 

1  5 

1 

1 

1  5 

4 

1 

3  0 

17.278 

1 

f 

12.722 

1 

f 

9.49 

1 

1 

9.49 

1 

1  8 

.  9  9 

5  0.00 

1 

1 

5  0.00 

4 

1 

1 

16.48 

• 

1 

2  2.39 

4 

1 

f 

•  f 

N  R 

( 

t 

1  6 

1 

1 

0 

1 

1 

1  6 

9.2152 

f 

1 

6.7  8  4  8 

4 

1 

10.13 

1 

• 

0.00 

4 

1 

1  0 

.  1  3 

100.00 

1 

1 

0.00 

1 

1 

1 

1 

17.58 

( 

1 

0.00 

4 

1 

> 

f 

Total 

9  1 

6  7 

15  8 

5  7 

5  9  4  2 

4  1 

10  0  .00 

COST 

OP  STSTKHS 

URDKR  OBVBLOPHBRT 

3  T  AT 

I  S  T  (  C  S 

FOR 

TABLB 

OP  COST 

BY  GROUP 

S  t  a  t 

i  s  t  i  c 

OP 

V  a 

1  u  e 

P  r  o  b 

Chi- 

Square 

5 

2  6  . 

4  15 

0.000 

Lika 

1  i  h  o  0  d 

Rati 

o  Chi 

-Square 

5 

3  2  . 

3  0  3 

0.000 

M  a  n  t 

el  H  a  e 

n 

s  X  e  1 

Chi- 

Square 

1 

0  . 

0  16 

0.899 

Phi 

C  a  e  f  t  i 

c 

lent 

0  . 

4  0  9 

C  o  n  t 

i  n  g  •  n  c 

y 

Coe 

f  f  i  c  i 

•  n  t 

0  . 

3  7  8 

C  c  a  m 

e  r  '  »  V 

0  . 

4  0  9 

Samp 

1  e  Six 

• 

-  1 

5  8 
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Table  1-5.  Lines  of  Code  in  Systems  Under  Development 

TABLE  OF  LOG  BY  GROUP 
LOG  GROUP 


Fr«qu«ncr 

Bipactsd 


P  a  c  c  a  n  t 

« 

I 

Row  Pet 

1 

t 

Col  Pet 

1 

1 

C  0  N  T 

1  SAMPLE 

1 

1 

Total 

-  f 

¥  ~ 

-  -  _ 

-  _  _  - 

¥ 

.  9  9  9  K 

1 

1 

4 

1 

1 

4 

» 

1 

8 

1 

1 

4.6  0  7  6 

1 

1 

3.3924 

1 

1 

1 

2 

.  5  3 

1 

1 

2 

.  5  3 

1 

5 

.  0  6 

1 

1 

5  0 

.  0  0 

t 

1 

5  0 

.  0  0 

1 

1 

1 

1 

4 

.  4  0 

1 

1 

5 

.  9  7 

1 

1 

-  f 

_  _  _  _ 

-  -  _  . 

- 

-  ,  _ 

. 

«> 

1  M 

1 

( 

3  0 

t 

t 

1  5 

1 

1 

4  5 

1 

1 

2  5  . 

9  18 

1 

1 

1  9  . 

0  8  2 

1 

1 

1 

1  3 

.  9  9 

t 

( 

9 

.  4  9 

1 

1 

2  8 

.  4  8 

1 

1 

6  6 

.  6  7 

1 

1 

3  3 

.  3  3 

« 

I 

« 

3  2 

.  9  7 

1 

1 

2  2 

.  3  9 

» 

t 

-  ♦ 

_  .  .  . 

f  - 

-  -  - 

-  -  -  - 

9  .  9  R 

1 

1  4 

1 

1  4 

1 

1 

2  8 

1 

1  6  . 

12  7 

t 

t 

1  1  . 

8  7  3 

1 

t 

1 

8 

.  8  6 

1 

1 

8 

.  8  6 

1 

1 

1  7 

.  7  2 

1 

1 

S  0 

.  0  0 

1 

( 

5  0 

.  0  0 

1 

1 

1 

1 

1  5 

.  3  8 

1 

1 

2  0 

.  9  0 

« 

1 

- 

-  •  -  - 

¥  - 

-  -  . 

-  -  -  . 

¥ 

9  9  .  9  K 

1 

t 

2  1 

1 

i 

2  9 

1 

1 

5  0 

1 

1 

2  8  . 

7  9  7 

1 

1 

2  1  . 

2  0  3 

« 

1 

1 

1  3 

.  2  9 

1 

1 

1  8 

.  3  5 

1 

t 

3  1 

.  6  5 

1 

1 

4  2 

.  0  0 

1 

1 

S  8 

.  0  0 

• 

t 

1 

1 

2  3 

.  0  8 

( 

1 

4  3 

.  2  8 

( 

( 

-  ♦ 

-  -  -  - 

¥  - 

-  - 

¥ 

>500 

1 

t 

1  0 

1 

1 

4 

t 

1 

1  4 

t 

1 

8.0  6  3  3 

1 

» 

5.9367 

1 

1 

1 

t 

6 

.  3  3 

t 

1 

2 

.  5  3 

t 

1 

8 

.  8  6 

1 

1 

7  1 

.  4  3 

1 

J 

2  8 

.  5  7 

» 

i 

1 

1 

1  0 

.  9  9 

1 

1 

5 

.  9  7 

1 

1 

-  f 

.  -  _  - 

-  -  -  . 

¥  - 

_  -  - 

¥ 

N  R 

1 

i 

1  2 

1 

1 

1 

t 

1  3 

1 

> 

7.4  8  7  3 

t 

1 

5.5127 

* 

« 

1 

1 

7 

.  5  9 

1 

I 

0 

.  6  3 

1 

8 

.  2  3 

1 

1 

9  2 

.  3  1 

1 

1 

7 

.  6  9 

1 

t 

1 

1 

1  3 

1  9 

1 

1 

1 

.  4  9 

1 

1 

- 

-  -  _  - 

-  ,  -  . 

¥  - 

^  - 

-  -  -  - 

¥ 

Total 

9  1 

6  7 

15  8 

5  7 

5  9 

4  2 

4  1 

10  0 

LINKS  OF  GODS 
STATISTICS  FOR 
Statistic 


H  SYSTEMS  UNDER  DEVELOPMENT 
TABLE  OF  LOG  BY  CROUP 

DP  V  a  1  u  a 


C  h 

i 

- 

S  q  u  a 

t 

a 

5 

1  4 

8 

5  6 

L  i 

k 

• 

1  i  h  o 

o 

d 

Rati 

0 

Chi 

-  3 

q  u  a  r  a 

5 

1  6 

3 

5  0 

M  a 

n 

t 

a  1  -  H 

a 

• 

n 

a  1  a  1 

C  h  i  - 

8  q 

u  a  I  a 

1 

0 

6 

6  1 

P  h 

1 

Coat 

t 

i 

c 

i  a  n  t 

0 

3 

0  7 

C  0 

n 

t 

i  n  q  a 

n 

c 

y 

Cos 

1 

t  i  c  i 

•  n 

t 

0 

2 

9  3 

C  c 

a 

m 

a  c  '  3 

V 

0 

3 

0  7 

8  a 

m 

P 

1  a  S 

i 

a 

• 

=  1 

5 

8 

P  c  o  b 


0.011 

0.006 

0.416 
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Table  1-6.  Organizational  Use  of  CASE  Tools 

TABL8  OF  ORGUSB  BY  GROUP 


ORG  IS  CROUP 


F  t  a  q  u  a 

n  c  y 

B  1  p  a  c  t 

•  d 

P  a  t  c  a  n 

t 

Ron  Pc 

t 

Col  Pc 

t 

COHT 

( 

1 

SAMPLB 

< 

1 

Total 

_ 

f 

_  .  _  _ 

_  _  _  _ 

f 

- - - 

»  -  -  - 

4- 

N  0 

4  1 

1 

1 

3  2 

t 

t 

7  3 

4  2  . 

0  4  4 

> 

1 

3  0  . 

9  5  6 

t 

1 

2  5 

.  9  5 

1 

1 

2  0 

.  2  5 

1 

1 

4  6.20 

5  6 

.  1  6 

1 

1 

4  3 

.  8  4 

1 

1 

4  5 

.  0  5 

1 

t 

4  7 

.  7  6 

1 

1 

_  -  - 

f 

- -  _ 

-  -  -  , 

4- 

- - - 

- - 

4- 

N  R 

1  6 

1 

t 

0 

1 

1 

1  6 

9.2152 

1 

6.7  8  4  8 

1 

1 

1  0 

.  1  3 

( 

1 

0 

.  0  0 

1 

1 

10.13 

10  0 

.  0  0 

t 

0 

.  0  0 

1 

1 

1  7 

.  5  8 

1 

1 

0 

.  0  0 

1 

— 

— 

- - 

f 

— 

- - 

f 

Y  B  S 

3  4 

1 

1 

3  5 

1 

i 

6  9 

3  9  . 

7  4  1 

1 

1 

2  9  . 

2  5  9 

1 

1 

2  1 

.  5  2 

1 

1 

2  2 

.  1  5 

« 

1 

4  3.67 

4  9 

.  2  8 

t 

1 

5  0 

.  7  2 

1 

1 

3  7 

.  3  6 

1 

1 

5  2 

.  2  4 

* 

1 

_  .  _ 

f 

-  •  -  - 

-  -  -  . 

f 

_  .  -  - 

,  _  -  - 

4- 

Total 

9  1 

6  7 

15  8 

5  7 

5  9 

4  2 

4  1 

10  0 

ST\TI8TICS  FOR  TABLK  OF  ORGOSS  BY  GROUP 


Statistic 


OF  V  a  1 ua 


Ciii-Bquaca  2  IJ.797 
Likalihood  Ratio  Chi-Squat#  2  19.648 
Mantal-Haanssal  Chi-Squat#  1  0.632 
FhiCoafficivnt  0.296 
Contingancy  CooEticiant  0.283 
Ciamat'sV  0.296 
Sampla  Sis#  =  158 


P  t  o  b 


0  .  0  0  I 
0.000 
0.426 
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Table  1-7.  Personal  Use  of  CASE  Tools 


TABLE  OF  ORGUSB  BY  GROUP 
ORGUSB  GROUP 


Fiequvncy 

Bzpactvd 

P  «  I  c  a  n  t 

Row  Pet 

Col  Pet 

COKT 

1 

1 

SAMPLE 

1 

1 

Total 

f 

4- 

H  0 

4  1 

1 

1 

3  2 

1 

1 

7  3 

42.044 

1 

1 

30.956 

1 

1 

2  5.95 

1 

1 

2  0.25 

1 

1 

4  6.20 

5  6.16 

1 

1 

4  3.84 

* 

1 

4  5.05 

« 

t 

4  7.76 

« 

( 

f 

f 

¥ 

H  R 

1  6 

1 

1 

0 

1 

1 

1  6 

9.2152 

1 

I 

6.7  8  4  8 

* 

1 

10.13 

1 

i 

0.00 

1 

t 

10.13 

10  0  .00 

1 

1 

0.00 

4 

1 

17.58 

1 

1 

0.00 

1 

t 

f 

- -  - 

f 

4* 

Y  B  8 

3  4 

1 

1 

3  5 

1 

t 

6  9 

3  9.7  4  1 

1 

1 

2  9.2  5  9 

1 

1 

2  1.52 

t 

i 

2  2.15 

1 

1 

4  3.67 

4  9.28 

t 

1 

5  0.72 

t 

< 

3  7.36 

1 

> 

5  2.24 

1 

t 

f 

-  *  -  - - -  ' 

4- 

Total 

9  1 

6  7 

15  8 

5  7 

5  9  4  2 

4  1 

100.00 

STATISTICS 

FOR 

TABLE 

OF  ORGUSB 

B  Y 

GROUP 

Statistic 

DF 

Value 

P  c  0  b 

Chi-Squara 

2 

1  3  . 

7  9  7 

0.001 

Likal ihood 

Rat 

i 

0  Chi 

-  S  q  u  a  t  a 

2 

1  9  . 

6  4  8 

0.000 

Mantal -Haa 

n  s  s  « 

1 

Chi- 

Square 

1 

0  . 

6  3  2 

0.426 

Phi  Coaffi 

c 

i  •  n 

t 

0  . 

2  9  6 

Continqanc 

I 

C  o 

• 

£  C  i  c  i 

ant 

0  . 

2  8  3 

Ciamac's  V 

0  . 

2  9  6 

Sampla  Six 

• 

- 

1 

5  8 
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Table  1-8.  Initial  Cost  of  Using  CASE  Tools 

TAB  IS  or  INITC08T  BY  GROUP 
ISITC08T  GROUP 


Chi-8quar«  6  6.249  0.396 

Likalitaood  Ratio  Chi-3quaca  6  6.423  0.378 

MantalHaanaial  Cbl-8quaio  1  0.630  0.427 

PhlCoatClciant  0.199 

Contingancr  Coofficiant  0.19S 

Cramac'aV  0.199 

8amplo  8iio  138 

MARRING:  50%  of  tho  colls  havo  oapoctod  counts  loss 

than  5.  Chi-8quacs  mar  nut  bo  a  valid  tost. 
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Table  1-9.  Recurring  Cost  of  Using  CASE  Tools 

TABLE  OF  RBCCOST  BY  GROUP 


RBCC08T  GROUP 


Pc«<iu«acy  1 
Bzp«ct»d  1 
P  •  c  c  •  n  t  1 

Row  Pet  i 

Col  Pet  1 

C  0  H  T 

t 

t 

SAMPLE 

4 

1 

Total 

- - 

.  9  9  »K  j 

1  7 

f 

1 

1 

1  3 

¥ 

* 

1 

3  0 

t 

1 

1 

1 

17.278 

10.76 

1 

1 

1 

1 

12.722 

8.23 

( 

4 

1 

4 

1  8 

9  9 

1 

1 

1 

* 

- - f 

4  .  9K  1 

S  6  .  6  7 
18.68 

3 

1 

t 

1 

1 

f 

1 

1 

4  3.33 
19.40 

3 

1 

f 

1 

1 

f 

4 

1 

6 

1 

1 

1 

1 

3.4557 

1  .  90 

1 

t 

1 

1 

2.5  4  4  3 

1  .  90 

4 

1 

4 

1 

3 

8  0 

1 

1 

1 

1 

4  9  .  9  K  1 

5  0.00 
3.30 

1 

1 

t 

1 

I 

f 

4 

t 

5  0.00 
4.48 

4 

4 

1 

4 

4 

f 

4 

1 

5 

1 

« 

1 

1 

2.8  7  9  7 
0.63 

i 

1 

1 

2.1203 

2.53 

1 

1 

4 

1 

3 

1  6 

t 

1 

1 

1 

5  0  K  1 

2  0.00 
1.10 

2 

4 

1 

4 

1 

f 

4 

1 

8  0.00 
5.97 

3 

4 

1 

4 

1 

¥ 

4 

4 

5 

1 

• 

1 

1 

2.8797 

1.27 

4 

1 

4 

1 

2.1203 

1.90 

1 

4 

4 

4 

3 

1  6 

1 

1 

1 

1 

- ...f 

9  .  9K  1 

4  0.00 
2.20 

2 

1 

1 

4 

1 

> 

1 

t 

6  0.00 

4.48 

4 

4 

I 

4 

1 

f 

4 

4 

6 

1 

t 

4 

3.4557 

1.27 

4 

1 

4 

1 

2.5  4  4  3 
2.53 

1 

4 

1 

3 

8  0 

< 

1 

! 

N  0  T  X  IT  0  H  1 

3  3.33 
2.20 

4  4 

4 

4 

1 

( 

•  f 

4 

i 

6  6.67 
5.97 

3  3 

4 

4 

4 

1 

'  f 

4 

1 

7  7 

1 

1 

1 

1 

44.348 

2  7.85 

4 

1 

4 

4 

3  2.6  5  2 

2  0.89 

4 

4 

4 

4 

4  8 

7  3 

1 

1 

t 

1 

H  R  ; 

5  7.14 

4  8.35 

2  2 

4 

1 

4 

4 

• 

1 

4 

4  2.86 

4  9.25 

7 

4 

4 

4 

4 

'  ¥ 

1 

1 

2  9 

1 

1 

1 

1 

16.703 

13.92 

1 

4 

( 

1 

12.297 

4.43 

1 

1 

4 

1 

1  8 

3  5 

1 

1 

t 

1 

---  -  —  -  --  f 

7  5.86 

2  4.18 

1 

4 

4 

4 

- 

2  4.14 
10.45 

4 

4 

• 

1 

-  ¥ 

Total 

9  1 

6  7 

15  8 

5  7.59 

4  2.41 

10  0  .00 

RECURRING 

COST 

OF  USING  CASE 

TOOLS 

STATISTICS 

FOR 

TABLE  OF  RECCOST  BY  GROUP 

Statistie 

D  F  Value 

P  c  o  b 

Chi- Square 
Likalihood  Rat 
Mantwl -Hawnaxw 
Phi  Coaftieian 
Continganey  Co 
Ccamar's  V 
Sample  Siia 

lo  Chi-Squaca 

1  Chi-Squaca 
t 

affieiant 

15  8 

6  9.094 

6  9.408 

1  1.636 

0.240 

0.233 

0.240 

0.168 

0.152 

0.201 

N  A  R  N  I  N  G  ; 

5  7  % 
than 

of  tha  ealls  have  azpaetad  eounts 
5.  Chi-Squaca  may  not  ha  a  valid 

lass 
t  a  s  t  . 

I-IO 


Table  I-IO.  First  Exposure  to  CASE  Tools 


TABLE  OF  EXPOSE  BY  CROUP 
EXPOSE  GROUP 


F  c  •  q  u  • 

n  c  7 

E  s  p  •  c  t 

•  d 

P  ■  r  c  •  n 

t 

Row  Pc 

t 

Col  Pc 

t 

C  OR  T 

1 

1 

SAMPLE 

1 

Total 

-  _  _ 

_  -  -  - 

- - 

¥ 

- - - 

¥ 

JOB 

2  1 

i 

1 

2  3 

1 

4  4 

2  S  . 

3  4  2 

1 

1 

X  8  .  6  5  8 

1 

1  3 

.  2  9 

1 

f 

X  4  .  5  6 

1 

2  7 

8  5 

4  7 

.  7  3 

1 

1 

5  2.27 

1 

2  3 

.  0  8 

1 

1 

3  4.33 

• 

1 

—  _ 

> 

-  -  -  _ 

_  -  -  - 

¥ 

¥ 

MAC/JOUR 

2  3 

t 

1 

X  X 

1 

3  4 

1  9  . 

5  8  2 

» 

1 

X  4  .  4  X  8 

* 

1 

1  4 

.  5  6 

1 

1 

6.96 

1 

1 

2  X 

5  2 

6  7 

.  6  5 

1 

I 

3  2.35 

1 

1 

2  S 

.  2  7 

1 

1 

X  6  .  4  2 

1 

1 

------ 

-  -  _ 

¥ 

_  -  -  - 

-  -  -  - 

¥ 

¥ 

N  R 

1  9 

« 

1 

4 

1 

2  3 

1  3  . 

2  4  7 

1 

1 

9.7  5  3  2 

1 

1  2 

.  0  3 

1 

1 

2.53 

1 

X  4 

5  6 

8  2 

.  6  1 

1 

1 

X  7  .  3  9 

1 

2  0 

.  8  8 

t 

1 

5.97 

1 

1 

-  -  - 

¥ 

-  -  -  - 

-  -  -  - 

¥ 

¥ 

OTHER 

1  2 

t 

t 

7 

1 

t 

X  9 

1  0  . 

9  4  3 

1 

« 

8.057 

1 

7 

.  5  9 

1 

1 

4.43 

t 

X  2 

0  3 

6  3 

.  1  6 

1 

t 

3  6.84 

* 

t 

1  3 

.  X  9 

t 

1 

X  0  .  4  5 

1 

1 

-  -  - 

¥ 

-  -  -  - 

f 

- - - - - 

SCHOOL 

1  6 

t 

1 

2  2 

• 

t 

3  8 

2  1  . 

8  8  8 

« 

t 

X  6  .  X  X  4 

1 

1  0 

.  1  3 

1 

1 

X  3  .  9  2 

1 

2  4 

0  5 

4  2 

.  X  X 

1 

1 

5  7.89 

1 

1  7 

.  5  8 

1 

1 

3  2.84 

* 

t 

-  -  - 

¥ 

-  -  -  - 

-  ,  - 

■ 

Total 

9  X 

6  7 

X  5  8 

5  7.59 

4  2  .  4  X 

X  0  0  .  0  0 

FIRST  EXPOSURE 

TO  CASE  TOOLS 

STATISTICS  FOR 

TABLE  OF  EXPOSE 

B  Y 

CROUP 

Statiatic 

DF 

Value 

P  r  o  b 

Chl-Squaia 

4 

X  3  . 

0  2  7 

0  .  0  X  X 

Likalibood  Rat 

io  Chi-Squaca 

4 

X  3  . 

6  7  3 

0.008 

Mantal -Haansxa 

1  Chi-Square 

X 

0  . 

3  2  X 

0  .  5  7  X 

Phi  Coaffician 

t 

0  . 

2  8  7 

Contingancr  Co 

efficient 

0  . 

2  7  6 

Ccamac's  V 

0  . 

2  8  7 

Sampla  Siia  = 

X  5  8 

I-ll 


Table  I-ll.  Length  (Years)  Using  CASE  Tools 

TABLB  or  LBNGTH  BY  GROUP 

LKNGTH  GROUP 

Fraquvncp  | 

Bspactad  I 
P  •  I  c  •  n  t  I 
Row  Pet  I 


Col  Pet 

1 

1 

C  0  R  T 

1 

1 

SAMPLE 

1 

1 

Total 

-  f 

¥ 

¥ 

1  -  2  Y  R  S 

i 

1 

8 

1 

1 

1  4 

1 

1 

2  2 

1 

1 

12.671 

1 

1 

9.3  2  9  1 

1 

1 

1 

1 

5.06 

1 

1 

8.86 

« 

1 

13.92 

1 

1 

3  6.36 

1 

1 

6  3.64 

1 

t 

1 

1 

8.79 

1 

1 

2  0.90 

( 

1 

'  f 

¥ 

♦ 

3  -  4  Y  R  S 

t 

1 

3 

1 

( 

4 

1 

1 

7 

I 

1 

4.0316 

1 

1 

2.9  6  8  4 

4 

1 

t 

1 

1.90 

1 

1 

2.53 

1 

1 

4.43 

1 

I 

4  2.86 

1 

1 

5  7.14 

1 

1 

1 

1 

3.30 

1 

5.97 

1 

1 

-  f 

¥ 

f 

<  1  Y  R 

t 

1 

3  1 

t 

1 

3  2 

1 

1 

6  3 

1 

1 

3  6.2  8  5 

1 

1 

2  6.715 

1 

1 

1 

1 

19.62 

< 

t 

2  0.25 

1 

1 

3  9.87 

1 

1 

4  9.21 

t 

1 

5  0.79 

t 

1 

1 

1 

3  4.07 

1 

1 

4  7.76 

1 

t 

-  f 

¥ 

¥ 

>  4  Y  R  8 

1 

1 

5 

i 

t 

4 

4 

1 

9 

1 

1 

5.1835 

t 

1 

3.8165 

4 

1 

1 

3.16 

1 

1 

2.53 

4 

1 

5.70 

i 

1 

5  5.56 

1 

1 

4  4.44 

4 

1 

t 

( 

5.49 

1 

1 

5.97 

1 

1 

-  f 

¥ 

f 

H  R 

1 

1 

4  4 

1 

t 

1  3 

4 

1 

5  7 

t 

( 

32.829 

t 

< 

2  4.171 

4 

1 

1 

t 

2  7.85 

1 

1 

8.23 

1 

1 

3  6.08 

1 

1 

7  7.19 

t 

( 

2  2.81 

4 

t 

1 

1 

4  8.35 

1 

1 

19.40 

1 

1 

-  ¥ 

f 

¥ 

Total 

9  1 

6  7 

15  8 

5  7 

5  9  4  2 

4  1 

10  0 

LBHGTH  (YEARS)  USING  CASK  TOOLS 


STATISTICS 

FOR  TABLE 

OF  LENGTH 

B  Y 

GROUP 

Statistic 

DF 

V  a 

1  u  • 

P  t  o  b 

Chi-Squaro 

4 

1  5  . 

4  7  7 

0.004 

Likolihood 

Ratio  Chi 

-  S  q  u  a  r  a 

4 

1  6  . 

0  7  6 

0.003 

Hantol -Haan 

sxol  Chi- 

8  q  u  a  c  a 

1 

1  4  . 

3  9  1 

0.000 

Phi  Coaffic 

i  a  n  t 

0  . 

3  13 

ContinqancT 

Cooffici 

•  n  t 

0  . 

2  9  9 

Cramar's  V 

0  . 

3  13 

Sampl*  Sliw  ISS 

MARNING:  30%  of  tha  calls  hava  aspactad  counts  loss 

than  5.  Chi-Squaca  mar  not  ba  a  valid  tost. 
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Table  1-12.  Errors/KLOC  w/o  Using  CASE  Tools 

TABLE  OF  NUHKLOC  BY  GROUP 

RUMKLOC  CROUP 

Fraquvncr  I 
Bsp«ct«d  I 
P  •  r  c  •  n  t  ! 

R  o  H  Pet  I 


Col  Pet 

t 

I 

COST 

1 

1 

SAMPLE 

1 

1 

Total 

-  f 

¥ 

1  0  -  2  5  E 

t 

1 

6 

1 

1 

1  4 

1 

1 

2  0 

t 

I  1  .  S  I  9 

1 

t 

8.481 

1 

1 

1 

3.80 

1 

i 

8.86 

1 

1 

12.66 

1 

t 

3  0.00 

1 

1 

7  0.00 

1 

1 

I 

1 

6  .  S  9 

1 

1 

2  0.90 

$ 

1 

-  ¥ 

♦ 

¥ 

2  6  -  S  0  E 

1 

1 

4 

1 

1 

7 

1 

1 

1  1 

t 

1 

S  .  3  3  S  4 

1 

1 

4.6  6  4  6 

1 

1 

1 

1 

2  .  S  3 

1 

1 

4.43 

1 

6.96 

1 

1 

3  6.36 

1 

( 

6  3.64 

1 

1 

1 

1 

4.40 

« 

1 

10.45 

1 

1 

-  ♦ 

f 

f 

5  1  -  1  0  0  E 

1 

1 

3 

« 

1 

7 

t 

1 

1  0 

1 

1 

5.7  5  9  5 

( 

1 

4.2405 

1 

1 

1 

1.90 

t 

1 

4.43 

4 

< 

6.33 

1 

3  0.00 

( 

7  0.00 

< 

1 

1 

1 

3.30 

t 

• 

10.45 

1 

1 

-  f 

f 

<  1  0  E 

1 

1 

1  3 

t 

1 

1  8 

1 

1 

3  1 

1 

1 

17.854 

1 

1 

13.146 

• 

1 

< 

8.23 

1 

1 

11.39 

t 

19.62 

t 

1 

4  1.94 

1 

( 

5  8.06 

1 

1 

1 

14.29 

I 

1 

2  6.87 

« 

1 

-  ♦ 

f 

¥ 

>  1  0  0  E 

1 

1 

2 

t 

1 

1 

1 

1 

3 

1 

1.7  2  7  8 

1 

( 

1  .  2  7  2  2 

1 

1 

1 

1 

1.27 

( 

1 

0.63 

« 

1 

1.90 

1 

6  6.67 

1 

t 

3  3.33 

1 

• 

I 

2.20 

1 

1 

1.49 

• 

t 

-  ♦ 

f 

f 

R  R 

1 

6  3 

1 

1 

2  0 

4 

1 

8  3 

1 

47.804 

t 

1 

3  5.196 

4 

1 

1 

3  9.87 

• 

1 

12.66 

1 

1 

5  2.53 

; 

7  5.90 

1 

< 

2  4.10 

1 

• 

6  9.23 

1 

2  9.85 

4 

( 

-  ♦ 

♦ 

♦ 

Total 

9  1 

6  7 

15  8 

5  7 

5  9  4  2 

4  1 

10  0 

ERRORS/KLOC  N/0  U8IRG  CASE  TOOLS 
STATISTICS  FOR  TABLE  OF  NUHKLOC  BY  CROUP 


8  t  a  t 

i 

a  t  i  e 

DF 

V  a  1  u 

a 

P  r 

Q  b 

Chi- 

8 

q  u  •  r  • 

5 

2 

5.98 

9 

0 

.  0 

0  0 

Lika 

1 

i  h  o  o  d 

R  a 

t 

i 

0  chi 

-  8  q  u 

a 

c 

a  5 

2 

6.65 

5 

0 

.  0 

0  0 

M  a  a  t 

• 

1  H  a  a  It 

m  1 

a 

1 

Chi- 

8  q  u  a 

t 

a 

1 

2 

2  .  8.6 

5 

0 

.  0 

0  0 

Phi 

C 

o  a  C  C  i  c 

i  • 

n 

t 

0.40 

6 

C  o  n  t 

i 

n  9  •  n  c  7 

c 

o 

a 

f  f  i  c  i 

ant 

0.37 

6 

Cram 

• 

c  '  a  V 

0.40 

6 

Samp 

1 

a  8  i  1  a 

= 

1 

5  8 

MARK 

I 

N  G  :  3 

3  % 

o 

f  t  h  a 

cal 

1 

a 

hava  aapa 

c 

tad 

c  o  u  n  t  a 

1  a 

a  a 

t 

h  a 

n 

5  .  C  h 

i  -  8  q 

u  a 

ca  may  aot 

b  a  a 

valid 

t  a 

a  t 
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Table  1-13.  Percentage  of  Errors  Credited  to  Req  Anal  w/o 
CASE  Tools 


TABLK  or  PBRBRR  BY  GROUP 

PBRKRR  GROUP 

Piaquvncr  | 

Ispactad 

P  a  I  c  a  n  t  I 

R  o  tf  Pet  ! 


Col  Pet 

1 

1 

C  0  N  T 

1 

1 

SAMPLE 

« 

1 

Total 

-  ♦ 

-  _ 

-  - 

-  >  _  - 

f 

-  - 

-  - 

-  -  «  - 

♦ 

1  0  -  2  5  E 

1 

5 

1 

1 

8 

1 

1 

1  3 

1 

1 

7 

.4  8  7  3 

I 

1 

5 

.5127 

4 

( 

1 

t 

3 

.  1  6 

1 

1 

5 

.  0  6 

1 

1 

8.23 

1 

1 

3  8 

.  4  6 

( 

( 

6  1 

.  5  4 

1 

1 

1 

1 

5 

.  4  9 

1 

1 

1  1 

.  9  4 

4 

1 

-  ♦ 

-  - 

-  - 

-  ,  _  - 

¥ 

-  - 

-  - 

_  -  -  - 

f 

2  5  -  5  0  E 

1 

1 

7 

1 

1 

8 

1 

1 

1  5 

1 

1 

8 

.6  3  9  2 

1 

1 

6 

.3  6  0  8 

4 

1 

1 

4 

.  4  3 

1 

I 

5 

.  0  6 

4 

1 

9.49 

1 

1 

4  6 

.  6  7 

1 

1 

5  3 

.  3  3 

4 

1 

t 

1 

7 

.  6  9 

1 

t 

1  1 

.  9  4 

4 

1 

f 

-  - 

-  - 

-  - - 

f 

-  - 

-  - 

_  _  -  - 

¥ 

5  -  1  0  E 

1 

1 

3 

4 

1 

1  2 

4 

1 

1  5 

t 

1 

8 

.6  3  9  2 

4 

1 

6 

.3  6  0  8 

4 

1 

f 

1 

1 

.  90 

1 

4 

7 

.  5  9 

1 

1 

9.49 

i 

2  0 

.  0  0 

4 

( 

8  0 

.  0  0 

f 

1 

1 

1 

3 

.  3  0 

4 

1 

1  7 

.  9  1 

4 

1 

- 

-  .. 

-  - 

-  ,  -  - 

♦ 

-  - 

-  - 

-  -  -  > 

«- 

<58 

t 

1 

1  2 

4 

( 

1  5 

t 

4 

2  7 

1 

1 

1  5  . 

5  5  1 

4 

1 

1  1  . 

4  4  9 

1 

1 

1 

t 

7 

.  5  9 

4 

4 

9 

.  4  9 

4 

1 

17.09 

1 

4  4 

.  4  4 

4 

1 

5  5 

.  5  6 

4 

1 

t 

1 

1  3 

.  1  9 

4 

1 

2  2 

.  3  9 

4 

1 

-  f 

-  - 

-  - 

-  -  .  - 

♦ 

-  - 

-  • 

-  -  -  , 

♦ 

>508 

t 

1 

4 

1 

2 

4 

1 

3 

1 

1 

1 

.7  2  7  8 

4 

4 

1 

.2  7  2  2 

4 

t 

t 

0 

.  6  3 

1 

4 

1 

.  2  7 

4 

1 

1.90 

1 

1 

3  3 

.  3  3 

4 

1 

6  6 

.  6  7 

4 

4 

1 

1 

1 

.  1  0 

1 

1 

2 

.  9  9 

1 

1 

-  ¥ 

-  - 

-  - 

_  -  - 

♦ 

-  - 

♦  • 

-  -  -  , 

N  R 

1 

1 

6  3 

4 

4 

2  2 

4 

4 

8  5 

1 

1 

4  8  . 

9  5  6 

1 

1 

3  6  . 

0  4  4 

4 

4 

1 

1 

3  9 

.  8  7 

4 

1 

1  3 

.  9  2 

4 

1 

5  3.80 

t 

t 

7  4 

.  1  2 

4 

2  5 

.  8  8 

4 

1 

• 

» 

6  9 

.  2  3 

4 

1 

3  2 

.  8  4 

4 

4 

-  f 

-  . 

-  - 

_  -  - 

¥ 

-  - 

-  - 

. 

Total 

9  1 

6  7 

15  8 

5  7 

5  9 

4  2 

4  1 

10  0 

PBRCBRTAGB  OF  ERRORS  CREDITED  TO  REQ  ANAL  N/0  CASE  TOOLS 

STATISTICS  FOR  TABLE  OP  PERERR  BY  CROUP 

Statistic  DF  Valua  Piob 


Chi-Squaia  5  2  3  .  i  9  9  0.000 

Likalihood  Ratio  Chi-Squaca  -5  24.187  0.000 

MantalHaanssal  Cbi-Squaia  1  16.054  0.000 

PhiCoatticiant  0.386 

Continqancr  Coafficiant  0.360 

Ccamac'sv  0.386 

Sampla  Sisa  158 
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Table  1-14.  Errors/KLOC  Using  CASE  Tools 

TABLE  or  NUMRLOC  BY  GROUP 

HUMKLOC  GROUP 

Fiaquancr  i 
Sspactad  ; 


P  •  r  c  •  n  t 
Row  Pet 


Col  Pet 

1 

1 

CORT 

1 

SAMPLE 

1 

1 

Total 

~  f 

-  - 

— 

_  _  _  - 

f 

_  - - 

- - - 

f 

1  0  -  2  5  K 

( 

1 

2 

1 

1 

7 

( 

t 

9 

1 

( 

S 

.18  3  5 

1 

1 

3.8165 

1 

1 

1 

1 

1 

.  2  7 

1 

1 

4 

.  4  3 

1 

1 

5 

.  7  0 

1 

1 

2  2 

.  2  2 

t 

1 

7  7 

.  7  8 

1 

1 

t 

2 

.  2  0 

1 

1 

1  0 

.  4  5 

( 

1 

-  f 

-  - 

— 

_  _  _  - 

f 

- -  - 

- - - 

2  6  -  5  0  B 

1 

1 

2 

1 

1 

3 

1 

t 

5 

1 

1 

2 

.8  7  9  7 

1 

1 

2.1203 

t 

1 

1 

1 

1 

.  2  7 

t 

1 

1 

.  9  0 

1 

1 

3 

.  1  6 

t 

i 

4  0 

.  0  0 

1 

t 

6  0 

.  0  0 

1 

1 

1 

1 

2 

.  2  0 

1 

1 

4 

.  4  8 

1 

1 

-  f 

-  - 

-  - 

_  _  _  - 

f 

_  _  -  - 

- - 

f 

5  1  -  1  0  0  K 

1 

1 

1 

1 

1 

2 

• 

1 

3 

t 

1 

1 

.7  2  7  8 

1 

t 

1.2  7  2  2 

( 

1 

1 

t 

0 

.  6  3 

1 

i 

1 

.  2  7 

1 

1 

1 

.  90 

1 

1 

3  3 

.  3  3 

1 

1 

6  6 

.  6  7 

1 

1 

1 

1 

1 

.  1  0 

« 

t 

2 

.  9  9 

1 

1 

-  - 

-  f 

-  - 

-  - 

-  -  -  - 

-  -  -  - 

<  1  0  B 

1 

1 

5 

1 

i 

2  0 

t 

1 

2  5 

< 

1  4  . 

3  9  9 

1 

( 

1  0  . 

6  0  1 

1 

1 

t 

3 

.  1  6 

» 

1 

1  2 

.  6  6 

1 

1 

1  5 

.  8  2 

1 

2  0 

.  0  0 

1 

1 

8  0 

.  0  0 

* 

1 

1 

1 

5 

.  4  9 

1 

t 

2  9 

.  8  5 

1 

1 

-  f 

-  - 

-  - 

-  -  -  . 

f 

-  _  -  . 

n  R 

« 

1 

8  1 

1 

1 

3  5 

1 

1 

116 

1 

1 

6  6 

.  8  1 

( 

« 

4  9 

.  1  9 

1 

1 

1 

5  1 

.  2  7 

1 

t 

2  2 

.  1  5 

* 

1 

7  3 

.  4  2 

1 

6  9 

.  8  3 

1 

1 

3  0 

.  1  7 

1 

1 

1 

t 

8  9 

.  0  1 

* 

1 

5  2 

.  2  4 

1 

1 

-  ♦ 

-  - 

-  - 

.  -  - 

♦ 

-  -  -  - 

-  -  .  , 

«* 

Total 

9  1 

6  7 

15  8 

5  7 

5  9 

4  2 

4  1 

10  0  .00 

BRROR8/KLOC  UStRG  CASK  TOOLS 

STATISTICS  FOR  TABLE  OF  HUHKLOC  BY  CROUP 


Statlatie 

DF 

V  a  1  u  o 

P  r  o  b 

Chi-Squaio 

4 

2  7 

.542 

0.000 

Likolihood  Ratio  Chi 

-Squaco  4 

2  8 

.213 

0.000 

Mantol  Haonsaol  Chi' 

S  q  u  a  [  o  1 

1  5 

.257 

0.000 

Phi  Cooffieiont 

0 

.418 

Continqaner  Cooffici 

•  n  t 

0 

.385 

Ccamoc's  V 

0 

.418 

Sampla  Slio  158 

NARRIRG;  50%  of  tho 

eolla  havo  oxp 

•  C  t 

od  eounts 

lass 

thaa  S.  Chi-Squaco  mar  not  bo  a  valid  tost. 
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Table  1-15.  Percentage  of  Errors  Credited  to  Req  Anal  with 
CASE  Tools 

TABLE  OF  PBRBRR  BY  GROUP 

PBRBRR  GROUP 

Fiaquancr  I 
8<p«ct«d  I 
P  •  I  c  •  n  t  I 
Row  Pet  I 


Col  Pet 

1 

1 

CORT 

1 

1 

SAMPLE 

« 

1 

Total 

-  M 

-  - 

-  - 

- 

f 

-  ~ 

— 

-  -  -  - 

f 

1  0  -  2  5  B 

1 

I 

4 

1 

1 

9 

1 

1 

1  3 

1 

1 

7 

.4  8  7  3 

1 

1 

5 

.5127 

• 

t 

1 

1 

2 

.  5  3 

1 

1 

5 

.  7  0 

1 

1 

8 

.  2  3 

1 

1 

3  0 

.  7  7 

1 

1 

6  9 

.  2  3 

i 

1 

1 

1 

4 

.  4  0 

1 

1 

1  3 

.  4  3 

t 

1 

-  f 

-  - 

-  - 

-  -  -  - 

¥ 

— 

-  - 

- -  - 

¥ 

2  S  -  5  0  B 

t 

1 

4 

1 

i 

3 

1 

1 

7 

< 

1 

4 

.0316 

1 

1 

2 

.9  6  8  4 

1 

1 

1 

2 

.  5  3 

1 

1 

1 

.  9  0 

1 

1 

4 

.  4  3 

1 

1 

5  7 

.  1  4 

1 

t 

4  2 

.  8  6 

1 

1 

i 

1 

4 

.  4  0 

1 

1 

4 

.  4  8 

1 

1 

-  f 

-  - 

-  - 

-  ,  -  . 

-  - 

— 

-  _  _  . 

¥ 

5  -  1  0  B 

1 

t 

1 

1 

1 

5 

1 

1 

6 

1 

i 

3 

.4  5  5  7 

1 

t 

2 

.5  4  4  3 

1 

1 

1 

1 

0 

.  6  3 

1 

1 

3 

.  1  6 

• 

t 

3 

.  8  0 

1 

1 

1  6 

.  6  7 

1 

I 

8  3 

.  3  3 

t 

1 

1 

1 

1 

.  1  0 

1 

I 

7 

.  4  6 

1 

1 

-  ♦ 

-  » 

-  ^ 

-  -  -  - 

♦ 

-  - 

-  - 

-  -  -  - 

♦ 

<  S  B 

< 

4 

« 

1  7 

( 

I 

2  1 

1 

1 

I  2  . 

0  9  5 

1 

1 

8 

.9  0  5  1 

( 

f 

1 

2 

.  5  3 

i 

« 

1  0 

.  7  6 

1 

1 

1  3 

.  2  9 

1 

1 

1  9 

.  0  5 

1 

1 

8  0 

.  9  5 

t 

t 

f 

« 

4 

.  4  0 

t 

1 

2  5 

.  3  7 

« 

8 

-  ♦ 

-  - 

— 

-  -  .  - 

¥ 

-  - 

-  - 

-  -  -  - 

♦* 

>  5  0  B 

1 

i 

0 

i 

1 

2 

1 

1 

2 

1 

i 

1 

.15  19 

t 

1 

0 

.8  4  8  1 

t 

t 

1 

t 

0 

.  0  0 

< 

1 

1 

.  2  7 

* 

1 

1 

.  2  7 

1 

1 

0 

.  0  0 

t 

» 

10  0 

.  0  0 

1 

t 

t 

i 

0 

.  0  0 

1 

2 

.  9  9 

1 

1 

-  f 

-  , 

-  - 

-  -  -  . 

¥ 

-  - 

-  - 

-  -  -  , 

¥ 

R  R 

1 

1 

7  8 

t 

1 

3  1 

1 

1 

10  9 

1 

1 

<  2  . 

7  7  8 

1 

1 

4  6  . 

2  2  2 

1 

1 

1 

1 

4  9 

.  3  7 

1 

1 

1  9 

.  6  2 

1 

1 

6  8 

.  9  9 

1 

f 

7  1 

.  5  6 

1 

1 

2  8 

.  4  4 

t 

I 

t 

1 

8  5 

.  7  1 

1 

1 

4  6 

.  2  7 

1 

1 

-  f 

-  - 

-  - 

_  -  -  . 

¥ 

*  - 

— 

- - 

¥ 

Total 

9  1 

6  7 

15  8 

5  7 

5  9 

4  2 

4  1 

10  0 

PERCENTAGE 

OF  ERRORS 

CRBDITBD 

TO 

RBQ  ANAL  WITH 

CASE  TOOLS 

STATISTICS 

FOR  TABLB 

OF  PBRBRR 

B  T 

GROUP 

Statistic 

DF 

Value 

P  t  o  b 

Chi-Squats 

5 

3  2  . 

14  2 

0.000 

Likelihood 

Ratio  Chi 

-Squats 

5 

3  3  . 

7  4  9 

0.000 

Mantel -Hasnsisl  Chi- 

Squats 

1 

1  7  . 

10  9 

0.000 

Phi  Coeffic 

i  e  n  t 

0  . 

4  5  1 

ContinqencT 

Coeffici 

•  n  t 

0  . 

4  11 

Ccaoiet's  V 
Sample  Sise 

=  15  8 

0  . 

4  5  1 

WARRING:  S0%  of  tbo  cwlls  bavw  wxpoctod  counts  loss 

tbsn  5.  Chi-Squsrw  rnsr  not  bo  a  valid  tost. 
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Table  1-16.  Errors  Increased  with  Using  CASE  Tools 

TABLK  OF  COHKBNT  BY  GROUP 


COMHBRT 

Frvquancr 

tipactad 

P  •  c  c  •  n  t 
Row  Pet 


GROUP 


Col  Pet 

( 

1 

CORT 

1 

1 

SAHPLB 

1 

1 

Total 

f 

f 

f 

AGRBB 

1 

1 

2 

1 

0 

1 

2 

1 

1 

1.1519 

< 

1 

0.8  6  8  1 

t 

1 

1 

t 

1.27 

t 

1 

0.00 

1 

1 

1.27 

I 

1 

10  0  .00 

1 

1 

0.00 

« 

1 

1 

1 

2.20 

1 

1 

0.00 

« 

1 

f 

f 

• 

D I SACRBB 

I 

1 

1  1 

1 

1 

1  6 

1 

1 

2  7 

I 

i 

15.551 

1 

1 

11.669 

1 

1 

1 

1 

6.96 

1 

1 

10.13 

1 

1 

17.09 

1 

1 

6  0.76 

1 

I 

5  9.26 

1 

1 

t 

1 

12.09 

« 

1 

2  3.88 

1 

1 

f 

f 

f 

NB I TH  BR 

< 

1 

1  0 

1 

1 

2  3 

1 

1 

3  3 

1 

1 

19.006 

1 

t 

13.996 

1 

1 

1 

1 

6.33 

t 

1 

16.56 

1 

• 

2  0.89 

t 

1 

3  0.30 

t 

1 

6  9.70 

1 

1 

1 

1 

10.99 

1 

1 

3  6.33 

1 

f 

¥ 

¥ 

R  R 

1 

1 

6  0 

6 

1 

2  0 

• 

1 

8  0 

1 

1 

6  6.0  7  6 

1 

1 

3  3.9  2  6 

1 

1 

1 

1 

3  7.97 

1 

12.66 

1 

1 

5  0.63 

1 

7  5.00 

1 

1 

2  5.00 

1 

1 

1 

6  5.93 

< 

1 

2  9.85 

1 

• 

f 

8TR AGRBB 

1 

1 

0 

1 

1 

1 

1 

1 

1 

1 

1 

0.5  7  5  9 

• 

• 

0.6261 

$ 

1 

1 

0.00 

1 

0.63 

» 

1 

0.63 

( 

( 

0.00 

1 

1 

10  0  .00 

6 

1 

1 

1 

0.00 

1 

1 

1.69 

1 

f 

¥ 

STRD I  SAG 

1 

1 

8 

1 

1 

7 

1 

1 

1  5 

f 

1 

8.6  3  9  2 

1 

1 

6.3  6  0  8 

1 

» 

1 

5.06 

1 

6.63 

1 

1 

9.69 

1 

1 

5  3.33 

1 

( 

6  6.67 

t 

1 

t 

1 

8.79 

1 

10.65 

< 

1 

f 

Total 

9  1 

6  7 

15  8 

5  7 

5  9  6  2 

6  1 

10  0 

0  0 


BRRORS  1HCRBA8BD  WITH 
STATISTICS  FOR  TABLB 
Statiatic 


USING  CASB  TOOLS 
OF  COHHBHT  BY  GROUP 

DF  V  a  1  u  a 


P  r  o  b 


Chi-Squara 

Likalihood  Ratio  Cbi-Squara 
Mantal-Haanaial  Chl-Squara 
Phi  Coatticiant 
Continqancr  Coafficiant 
Cramar'a  V 
Sampla  Siia  158 


5 

2  6 

0  7  0 

0  . 

.000 

5 

2  7 

6  9  0 

0  . 

.000 

1 

3 

90  8 

0  . 

.068 

0 

6  0  6 

0 

3  7  6 

0 

6  0  6 

WARN  I  RG  : 


33%  of  tba  colls  hava  ospactod  counts  loss 
than  5.  Chi-Squara  mar  not  bo  a  valid  tost. 
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Table  1-17.  Productivity  Increased  with  Using  CASE  Tools 


TABLE  OF  COHHBIIT  BY  OROUP 

COMMENT  GROUP 

Pcaquancr  ! 

Espactad  ! 

P  a  c  c  a  a  t  ! 

Row  Pet  : 


Col  Pet  ! 

CONT 

1 

1 

SAMPLE 

t 

t 

Total 

- - - - f 

f 

¥ 

AGREE 

1  8 

1 

1 

1  5 

1 

I 

3  3 

1 

1 

19.006 

1 

t 

13.994 

4 

1 

1 

1 

11.39 

< 

1 

9.49 

1 

1 

2  0.89 

1 

1 

S  6  .  S  5 

« 

1 

4  5.45 

1 

1 

1 

1 

19.78 

1 

t 

2  2.39 

1 

1 

f 

DISAGREE 

1 

1 

< 

0 

4 

1 

1 

1 

0.5  7  5  9 

I 

1 

0.4241 

1 

1 

< 

1 

0.63 

t 

1 

0.00 

1 

1 

0.63 

1 

10  0  .00 

1 

0.00 

4 

1 

t 

I 

1.10 

4 

1 

0.00 

* 

1 

f 

¥ 

NEITHER 

1  3 

4 

1 

2  7 

1 

1 

4  0 

1 

1 

23.038 

1 

1 

16.962 

4 

1 

1 

1 

8.23 

1 

( 

17.09 

• 

1 

2  5.32 

1 

3  2.50 

4 

6  7.50 

4 

t 

1 

14.29 

1 

1 

4  0.30 

1 

1 

_  -  -  -  -f 

f 

¥ 

N  R  1 

5  9 

1 

1 

1  8 

4 

1 

7  7 

1 

1 

4  4.3  4  8 

1 

1 

32.652 

4 

1 

1 

1 

3  7.34 

1 

11.39 

1 

i 

4  8.73 

» 

1 

7  6.62 

4 

t 

2  3.38 

4 

1 

1 

1 

6  4.84 

1 

1 

2  6.87 

4 

1 

♦ 

STRAGRIE 

0 

1 

6 

4 

1 

6 

1 

1 

3.4557 

4 

1 

2.5  4  4  3 

4 

1 

« 

1 

0.00 

1 

1 

3.80 

4 

1 

3.80 

« 

1 

0.00 

1 

1 

10  0  .00 

( 

1 

t 

t 

0.00 

4 

( 

8.96 

4 

4 

f 

♦ 

STRD I  SAC  1 

0 

1 

1 

1 

4 

1 

1 

t 

1 

0.5  7  5  9 

4 

1 

0.4  2  4  1 

1 

» 

1 

1 

0.00 

1 

1 

0.63 

4 

1 

0.63 

1 

0.00 

4 

1 

100.00 

4 

1 

• 

1 

0.00 

1 

1 

1.49 

t 

1 

. . -f 

Total 

9  1 

¥ 

6  7 

*■ 

15  8 

5  7 

5  9  4  2 

4  1 

10  0 

PRODUCTIVITY  INCREASED  NITH  USING  CASE  TOOLS 
STATISTICS  FOR  TABLE  OF  COMMENT  BY  CROUP 


Statiatic 

DF 

V  a  1  u  a 

P  r  0  b 

Chi-Squara 

5 

3  2  . 

0  9  9 

0.000 

Likalihood  Ratio 

Chi 

-Squara  5 

3  5  . 

7  10 

0.000 

Mantal'Haanaial  Chi- 

8  q  u  a  I  a  1 

0  . 

9  8  3 

0.321 

Phi  Coatficiant 

0  . 

4  5  1 

Continganep  Coaff 

i  c  i 

•  n  t 

0  . 

4  11 

Craaiac's  V 

Sampla  Sita  =  158 

0  . 

4  5  1 

MARRING:  50%  of 

t  h  a 

calla  hava  aapa 

ctad  counts 

less 

than  5  . 

Chi-Squara  may  not 

b  • 

a  valid 

tost. 
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Table  1-18.  Development  Costs  Increased  Using  CASE  Tools 


TUBLB  OP  COMHBHT  BY  GROUP 


COMMBIIT  OROU  P 

Pc«qu«acr  i 
Bzpactacl  | 

P  •  c  c  •  a  t  I 

Row  Pet  I 


Col  Pet 

1 

1 

CONT 

1 

1 

SAMPLE 

1 

1 

Total 

¥ 

¥ 

AGREE 

1 

1 

5 

1 

1 

6 

1 

1 

1  1 

1 

1 

6.3  3  5  4 

1 

1 

4.6  6  4  6 

1 

1 

1 

1 

3.16 

t 

t 

3.80 

1 

1 

6 

.  9  6 

1 

i 

4  5.45 

1 

1 

5  4.55 

1 

1 

1 

1 

5.49 

1 

1 

8.96 

1 

I 

¥ 

f 

DISAGREE 

t 

1  1 

1 

1 

1  3 

1 

1 

2  4 

1 

13.823 

1 

1 

10.177 

1 

1 

1 

1 

6.96 

1 

1 

8.23 

* 

1 

1  5 

.  1  9 

t 

4  5.83 

t 

1 

5  4.17 

1 

1 

t 

1 

12.09 

1 

1 

19.40 

1 

1 

¥ 

¥ 

NEITHER 

1 

1 

1  4 

1 

1 

2  7 

1 

1 

4  1 

t 

2  3.614 

1 

1 

17.386 

1 

t 

1 

8.86 

1 

1 

17.09 

t 

1 

2  5 

.  9  5 

1 

1 

3  4.15 

• 

1 

6  5.85 

1 

1 

1 

1 

15.38 

1 

I 

4  0.30 

1 

1 

- 

f 

¥ 

¥ 

N  R 

1 

1 

6  0 

1 

I 

2  0 

t 

1 

80 

1 

4  6.0  7  6 

< 

3  3.9  2  4 

1 

1 

t 

1 

3  7.97 

1 

1 

12.66 

t 

1 

5  0 

.  6  3 

1 

1 

7  5.00 

1 

( 

2  5.00 

1 

1 

1 

1 

6  5.93 

1 

1 

2  9.85 

f 

1 

¥ 

¥ 

f 

8TRO I  SAG 

1 

1 

1 

1 

1 

1 

1 

1 

2 

1 

t 

1.15  19 

1 

1 

0.8  4  8  1 

t 

1 

( 

0.63 

1 

0.63 

1 

1 

.  2  7 

1 

5  0.00 

* 

5  0.00 

1 

1 

« 

« 

1.10 

• 

1 

1.49 

1 

1 

• 

¥ 

♦ 

Total 

9  1 

6  7 

15  8 

5  7 

5  9  4  2 

4  1 

10  0.00 

DBVBLOPMINT  COSTS  IRCRBA8B0  NITH  USING  CASK  TOOLS 


STATISTICS  POR  TABLE  OP  COMMENT  BY  CROUP 


Statiatle 

DP 

V  a  1  u  a 

P  I  o  b 

Chi ' Squaca 

4 

2  1 

.224 

0.000 

Likalihood  Ratio 

Chi 

-Squara  4 

2  1 

.722 

0.000 

Mantal  Haansial  Chi- 

S  q  u  a  c  a  1 

1  0 

.12  1 

0.001 

Phi  CoaEticiant 

0 

.367 

Continganer  Coaff 

i  c  i 

•  n  t 

0 

.344 

Ccamac'a  V 

0 

.367 

Sampla  Sisa  =  158 

MARRING:  30%  of 

t  h  • 

calla  hava 

a  z  p  a  e  t 

ad  eounts 

lass 

than  5  . 

Chi-Squaca  may 

not  b 

a  a  valid 

t  a  a  t  . 
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Table  1-19.  Project  Completion  Time  Increased  Using  CASE 
Tools 

TABLS  OF  COMMSNT  BY  GROUP 

COHNBNT  GROUP 

Fc«qu«ncr  I 
Bzpactad  I 
P  •  I  c  •  n  t  I 
Row  Pet  I 


Col  Pet 

1 

I 

C  0  N  T 

1  S  A  M  P  L  B 

1 

1 

Total 

f 

_  -  -  - 

- - - 

f  - 

_  _  _ 

- - 

- 

AG  R  B  B 

1 

4 

t 

1 

4 

1 

1 

8 

1 

1 

4.6076 

1 

1 

3.3  9  2  4 

1 

1 

1 

1 

2 

.  5  3 

1 

t 

2 

.  5  3 

1 

1 

5.06 

1 

1 

S  0 

.  0  0 

t 

1 

5  0 

.  0  0 

1 

1 

1 

1 

4 

.  4  0 

1 

1 

5 

.  9  7 

1 

1 

f 

_  -  _  _ 

-  -  -  . 

t  - 

— 

-  -  -  . 

¥ 

OISAGRBB 

1 

1 

1  4 

1 

i 

1  8 

1 

1 

3  2 

I 

I 

1  8 

.  4  3 

1 

I 

1  3 

.  5  7 

1 

1 

1 

1 

8 

.  8  6 

1 

1 

1  1 

.  3  9 

I 

1 

2  0.25 

1 

1 

4  3 

.  7  5 

1 

1 

5  6 

.  2  5 

1 

1 

1 

1 

1  5 

.  3  8 

1 

1 

2  6 

.  8  7 

( 

1 

f 

-  _  ,  _ 

-  ,  -  - 

- 

- - 

- - - 

¥ 

R  B  I  T  H  B  R 

1 
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